2580 migrate schemas to schemas 20 #2603

JoeCap08055 · 2025-09-15T18:53:26Z

Goal

The goal of this PR is to update the types, extrinsics, and runtime APIs for the Schemas pallet to conform to the new design

Closes #2580

Checklist

JoeCap08055 · 2025-09-15T21:01:07Z

common/helpers/Cargo.toml

Removed packages that were only being used for schemas pallet RPCs that are now removed

JoeCap08055 · 2025-09-15T21:03:52Z

common/helpers/src/avro.rs

None of this was being used except for one custom RPC in the schemas pallet that allows a user to validate an Avro schema. However, since the chain itself does not do any validation of schema models (aside from size, and some extremely basic Frequency-specific Parquet validation), this seems very unnecessary. The onus is therefore on both the owner of a schema and Governance to validate its correctness before uploading to the chain.

JoeCap08055 · 2025-09-15T21:54:09Z

pallets/messages/src/rpc/src/lib.rs

I actually see no need for the messages pallet to have a custom RPC identical to the one we just removed from the schemas pallet. Upcoming PR may remove this one as well...

JoeCap08055 · 2025-09-15T21:54:52Z

pallets/messages/src/runtime-api/src/lib.rs


 		/// Retrieve a schema by id
-		fn get_schema_by_id(schema_id: SchemaId) -> Option<SchemaResponse>;
+		fn get_schema_by_id(schema_id: SchemaId) -> Option<SchemaResponseV2>;


same as above; this seems unnecessarily duplicative; the schemas runtime API already has this method

This doesn't looks like backwards compatible. I think maybe we want to keep this and add a new function and then remove the old one later

JoeCap08055 · 2025-09-15T21:56:57Z

pallets/schemas/src/rpc/Cargo.toml

All custom RPCs for schemas pallet have been removed, so there is no RPC client or server for this pallet.

JoeCap08055 · 2025-09-15T21:58:22Z

pallets/schemas/src/lib.rs

 		StorageMap<_, Twox64Concat, IntentGroupId, IntentGroup<T>, OptionQuery>;

 	#[pallet::genesis_config]
 	pub struct GenesisConfig<T: Config> {


ALL state storage is now settable by the genesis config; this makes it much easier to set up tests.

pallets/schemas/src/lib.rs

codecov · 2025-09-15T22:21:30Z

Codecov Report

❌ Patch coverage is 42.81525% with 195 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
pallets/schemas/src/migration/v5.rs	0.00%	102 Missing ⚠️
pallets/schemas/src/lib.rs	69.72%	56 Missing ⚠️
common/primitives/src/schema.rs	29.41%	24 Missing ⚠️
pallets/schemas/src/types.rs	28.57%	10 Missing ⚠️
pallets/schemas/src/migration/v4/mod.rs	0.00%	3 Missing ⚠️

Files with missing lines	Coverage Δ
pallets/messages/src/rpc/src/lib.rs	`96.07% <100.00%> (ø)`
pallets/stateful-storage/src/lib.rs	`78.60% <100.00%> (ø)`
pallets/schemas/src/migration/v4/mod.rs	`0.00% <0.00%> (ø)`
pallets/schemas/src/types.rs	`84.04% <28.57%> (-1.53%)`	⬇️
common/primitives/src/schema.rs	`37.86% <29.41%> (-5.89%)`	⬇️
pallets/schemas/src/lib.rs	`80.10% <69.72%> (-10.37%)`	⬇️
pallets/schemas/src/migration/v5.rs	`0.00% <0.00%> (ø)`

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

pallets/stateful-storage/src/tests/mock.rs

scripts/init.sh

tools/genesis-data/getSchema.mjs

pallets/schemas/src/migration/v5.rs

saraswatpuneet · 2025-09-16T23:51:32Z

pallets/schemas/src/migration/v5.rs

+/// - Reads/Writes from checking/settings the on-chain storage version are accounted for
+pub type MigrateV4ToV5<T> = frame_support::migrations::VersionedMigration<
+	4, // The migration will only execute when the on-chain storage version is 4
+	5, // The on-chain storage version will be set to 5 after the migration is complete


this is cool

saraswatpuneet

gave a read through, tested try runtime locally so that checks out. Some nits and it would be nice to have a one migration test

shannonwells

Looks good to me - only non-blocking comments/questions/suggestions

I read through the changes
I verified that create_schema_v4 is filtered out of mainnet
I verified that creating_schema_via_governance checks for correct origin
I checked that the error conditions for the new extrinsics were already unit tested.
I did not look at: generated weights or migration code.

shannonwells · 2025-09-16T01:01:39Z

tools/genesis-data/getSchema.mjs

+//   },
+// };
+
+/// Schema genesis format


What's the plan for this commented-out code?

This is informational--since Wil originally developed this as a pure JS tool, I didn't feel like introducing the overhead to convert to Typescript, but I wanted to document the structure of the genesis config as defined by the Rust implementation.

That said, I'd be happy to convert to Typescript if we think that would be cleaner. It's just that it adds a bunch to the tooling required. But, since this is not something that runs in CI, that might be okay.

pallets/schemas/README.md

scripts/init.sh

shannonwells · 2025-09-17T00:51:15Z

pallets/schemas/src/lib.rs


 		/// a method to return all versions of a schema name with their schemaIds
 		/// Warning: Must only get called from RPC, since the number of DB accesses is not deterministic
 		pub fn get_schema_versions(schema_name: Vec<u8>) -> Option<Vec<SchemaVersionResponse>> {


suggestion: could make this only a crate-visible function to help communicate this intended restriction. It doesn't prevent its being called by other crate members, but it can't be pulled in outside of the crate that way.

Suggested change

pub fn get_schema_versions(schema_name: Vec<u8>) -> Option<Vec<SchemaVersionResponse>> {

pub(crate) fn get_schema_versions(schema_name: Vec<u8>) -> Option<Vec<SchemaVersionResponse>> {

I don't think that would that prevent it from being called by an extrinsic within the crate, though, would it? That's the main point here.

pallets/schemas/src/migration/v5.rs

Co-authored-by: Shannon Wells <shannonwells@users.noreply.github.com>

aramikm

Added some comments we should also create an issue about updating js/schemas library

aramikm · 2025-09-17T19:40:55Z

pallets/messages/src/runtime-api/src/lib.rs


 		/// Retrieve a schema by id
-		fn get_schema_by_id(schema_id: SchemaId) -> Option<SchemaResponse>;
+		fn get_schema_by_id(schema_id: SchemaId) -> Option<SchemaResponseV2>;


This doesn't looks like backwards compatible. I think maybe we want to keep this and add a new function and then remove the old one later

pallets/schemas/src/lib.rs

pallets/schemas/src/migration/v5.rs

Co-authored-by: Aramik <aramikm@gmail.com>

enddynayn

🚀 impressive work!

JoeCap08055 added 14 commits September 3, 2025 19:32

wip: schemas 2.0 extrinsic deprecations, genesis and migrations

e94a2e2

feat: clean up benchmarks, migrations

8dd516d

fix: format

b8b57fa

fix: broken unit tests

2b0a4f0

feat: update weights, fix benchmarks, tests

aac49b9

fix: format

b6b40d7

feat: wip genesis state and migrations

ee3b3de

fix: name length migration

18a227d

fix: genesis ser/deser

6cd2d58

fix: lint

31988ce

fix: lint

aa30a9f

remove RPCs, fix RuntimeApi compatibility, fix bug in name migration

3b2619e

cleanup

073e869

fix CI error

dc7c45e

JoeCap08055 commented Sep 15, 2025

View reviewed changes

common/helpers/Cargo.toml

Copy link

Collaborator Author

JoeCap08055 Sep 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed packages that were only being used for schemas pallet RPCs that are now removed

JoeCap08055 commented Sep 15, 2025

View reviewed changes

JoeCap08055 marked this pull request as ready for review September 15, 2025 21:06

JoeCap08055 requested review from demisx and wilwade as code owners September 15, 2025 21:06

JoeCap08055 commented Sep 15, 2025

View reviewed changes

pallets/schemas/src/lib.rs Outdated Show resolved Hide resolved

github-actions bot added metadata-changed Metadata has changed since the latest full release metadata-version-not-incremented Metadata has changed since the latest full release, but the version has not been incremented labels Sep 15, 2025

enddynayn reviewed Sep 16, 2025

View reviewed changes

pallets/stateful-storage/src/tests/mock.rs Show resolved Hide resolved

enddynayn reviewed Sep 16, 2025

View reviewed changes

scripts/init.sh Show resolved Hide resolved

enddynayn reviewed Sep 16, 2025

View reviewed changes

tools/genesis-data/getSchema.mjs Outdated Show resolved Hide resolved

saraswatpuneet reviewed Sep 16, 2025

View reviewed changes

pallets/schemas/src/migration/v5.rs Show resolved Hide resolved

saraswatpuneet reviewed Sep 16, 2025

View reviewed changes

saraswatpuneet approved these changes Sep 16, 2025

View reviewed changes

shannonwells approved these changes Sep 17, 2025

View reviewed changes

JoeCap08055 commented Sep 17, 2025

View reviewed changes

pallets/schemas/src/migration/v5.rs Outdated Show resolved Hide resolved

Apply suggestions from code review

2d8e748

Co-authored-by: Shannon Wells <shannonwells@users.noreply.github.com>

aramikm reviewed Sep 17, 2025

View reviewed changes

Apply suggestions from code review

70a9323

Co-authored-by: Aramik <aramikm@gmail.com>

fix: format

9167e6d

github-actions bot removed metadata-changed Metadata has changed since the latest full release metadata-version-not-incremented Metadata has changed since the latest full release, but the version has not been incremented labels Sep 18, 2025

enddynayn approved these changes Sep 18, 2025

View reviewed changes

github-actions bot added metadata-changed Metadata has changed since the latest full release metadata-version-not-incremented Metadata has changed since the latest full release, but the version has not been incremented labels Sep 18, 2025

fix: tests and updated targets for try-runtime

9468379

JoeCap08055 merged commit fb63aab into feat/schemas-permissions-development Sep 18, 2025
29 of 32 checks passed

JoeCap08055 deleted the 2580-migrate-schemas-to-schemas-20 branch September 18, 2025 20:31

JoeCap08055 mentioned this pull request Sep 22, 2025

Migrate Schemas to Schemas 2.0 #2580

Closed

5 tasks

	pub fn get_schema_versions(schema_name: Vec<u8>) -> Option<Vec<SchemaVersionResponse>> {
	pub(crate) fn get_schema_versions(schema_name: Vec<u8>) -> Option<Vec<SchemaVersionResponse>> {

2580 migrate schemas to schemas 20 #2603

2580 migrate schemas to schemas 20 #2603

Uh oh!

Conversation

JoeCap08055 commented Sep 15, 2025

Goal

Checklist

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

codecov bot commented Sep 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

saraswatpuneet left a comment

Choose a reason for hiding this comment

Uh oh!

shannonwells left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

aramikm left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

enddynayn left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

codecov bot commented Sep 15, 2025 •

edited

Loading