feat: add user-defined schema and migrations

[sadortun]

New Pull Request Checklist

Continuation of PR #7091 from @Moumouls

• I am not disclosing a vulnerability.
• I am creating this PR in reference to an issue.

Issue Description

Related issue: #7063

Approach

Provide an easy way to add and handle schema updates and migrations for Parse-Server.

Examples:

How to use it in for Parse-server initialization :
https://github.com/sadortun/GoPlan-app/blob/feat-holding-history/backend/src/server.ts#L73

How to define your schemas :
https://github.com/sadortun/GoPlan-app/tree/feat-holding-history/backend/src/Migrations/schemas

TODOs before merging

• Add better logging/error managment

....

• Add test cases
• Add entry to changelog
• Add changes to documentation (guides, repository pages, in-code descriptions)

[sadortun]

LMK if you think it's OK, ill write some docs to go with this.

[codecov]

Codecov Report

Merging #7418 (63422fc) into alpha (653d257) will decrease coverage by 0.12%.
The diff coverage is 86.32%.

@@            Coverage Diff             @@
##            alpha    #7418      +/-   ##
==========================================
- Coverage   93.99%   93.87%   -0.13%     
==========================================
  Files         181      183       +2     
  Lines       13382    13626     +244     
==========================================
+ Hits        12579    12792     +213     
- Misses        803      834      +31     

Impacted Files	Coverage Δ
src/Adapters/Storage/Postgres/PostgresClient.js	70.00% <0.00%> (ø)
src/Options/index.js	100.00% <ø> (ø)
src/SchemaMigrations/Migrations.js	0.00% <0.00%> (ø)
src/Config.js	89.57% <62.50%> (-3.82%)	⬇️
src/SchemaMigrations/DefinedSchemas.js	92.22% <92.22%> (ø)
...rc/Adapters/Storage/Mongo/MongoSchemaCollection.js	97.84% <100.00%> (+0.09%)	⬆️
src/Adapters/Storage/Mongo/MongoStorageAdapter.js	93.36% <100.00%> (+0.67%)	⬆️
...dapters/Storage/Postgres/PostgresStorageAdapter.js	95.45% <100.00%> (+0.08%)	⬆️
src/Controllers/SchemaController.js	97.36% <100.00%> (+<0.01%)	⬆️
src/Options/Definitions.js	100.00% <100.00%> (ø)
... and 8 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 653d257...63422fc. Read the comment docs.

[mtrezza]

I executed the test and some fail. Can you look into that?

[sadortun]

Seems that test also failed in #7091 And are all related to Postgre.

I'll try to have a look but I don't use this DB.

[Moumouls]

What is Migration seems to be a lot of duplicated code ?

I think that you need to re use the defined schema business logic and export the method through ParseServer Object, you also need to keep the trigger "beforeMigration" in the loop.

Also a nice approach could be to let the developer perform some migration later with an option preventAutoMigration: true, then Parse Server instance could return a callback performMigration(), may be something like server.migrate() to apply defined schema provided at startup, when he want (may be after some stuff)

Out of scope but later we can implement a simple endpoint on the Parse Dashboard to trigger the .migrate(), and a user can click on it many times without risks since my code is stateless 🙂

What do you think about this. So you only need to keep one code maintained and well tested. We just add some other way to trigger the migration (auto, lazy, API, or may be CLI)

@mtrezza if you have a suggestion here 🙂

[Moumouls]

The flaky test Expected undefined to equal Object({ username_1: Object({ username: 1 }), case_insensitive_username: Object({ username: 1 }), email_1: Object({ email: 1 }), case_insensitive_email: Object({ email: 1 }) }). is hard to fix, may be we need to update it or skip it, but it seems that some desync code into SchemaCache/DB Adapter lead to unpredictable result on default core indexes. I don't know why, but if i remember correctly the flaky test just appeared after the major refactor of schema cache.

[sadortun]

What is Migration seems to be a lot of duplicated code ?

Ahhhhhhhhhhhhh! Ok, when I implemented my migrations I followed the links to your test Repo

#7091 (comment)

I see not that it was probably outdated and replaced by DefinedSchema

Let me fix everything and I'll update this lol!

@Moumouls I'm a bit confused by your comment. Migratipns.js is just your MaleSchema/buildSchema file in your repo. I literally just moved it there.

As you mentioned in the other thread, your code is already working well and used in production. I'm only here to do the finishing touches and get it merged. I have no intention of rewriting everything you've done so far.

[Moumouls]

@sadortun here there was a misunderstanding i think ahha, i already migrated and adjusted the code from my repo to DefinedSchema, here the only things we need to finish is to allow a lazy trigger of migration and add option to prevent auto migration. Everything is already implemented and well tested

I can invite you to give a look to defined schema code 🙂 In the PR changes

[sadortun]

@sadortun here there was a misunderstanding i think ahha, i already migrated and adjusted the code from my repo to DefinedSchema, here the only things we need to finish is to allow a lazy trigger of migration and add option to prevent auto migration. Everything is already implemented and well tested

Yeah, I just saw that ! I eddited my previous comment like 29 seconds ago!

Sooooo, I've been using the wrong thing for like 6 months ;)

Let me fix this today :)

[sadortun]

For the lint issues, Can we enable this ? Or I need to remove all the usages of ?. ?

Experimental optional chaining (`?.`) usage. Optional chaining is an active early-stage feature proposal that may
change. You may opt in to using it anyway by putting `esproposal.optional_chaining=enable` into the `[options]` section
of your `.flowconfig`.

[mtrezza]

I'd rather not use it as long as we are maintaining a LTS branch with a Node version that does not support this. We have found with a recent PR that this causes additional effort when backporting commits (even if we it's unlikely that we'll backport this schema PR) and it causes branches to divert more. So I'd not use optional chaining across the codebase at the moment.

As for merging alpha, I already did it a few times. It should be up to date.

We just had another release today, could you please merged again?

You can see this here (if you can see this section):

[mtrezza]

For the record:
codecov/patch — 86.34% of diff hit (target 93.99%) --> this PR has 86% coverage
codecov/project — 93.89% (-0.11%) compared to 090350a --> merging this PR will decrease parse server coverage by 0.11%

I think that is an acceptable coverage to merge and we can increase coverage across the codebase in future PRs, such as #7654 as @dblythy already started.

So all that's left to merge is the lint fix and bringing this up-to-date with alpha branch.

[sadortun]

@mtrezza
✅ merged alpha
✅Fixed Flow errors

---

I think that is an acceptable coverage to merge and we can increase coverage across the codebase in future PRs

If you look at the report, most, if not all un-covered LOC relates to logging to the user about warnings or errors. I'm prety sure that 1-2 added tests would fix that. But it can wait 😅 !

[mtrezza]

Looks good! This was a long time coming, originally started in Dec 2020 it's been almost 1 year in the making. Amazing work everyone 🙌 Big thanks to @Moumouls for starting this, to @sadortun for refining and maturing it to get it ready for merge and to everyone who contributed along the way.

[parseplatformorg]

🎉 This change has been released in version 5.0.0-alpha.5

[sadortun]

@mtrezza thanks a lot for your time and support ! Hope this feature will help users get their apps into production faster !

[Moumouls]

Amazing, i'm so happy to see this feature finally joining the official repo. It was a long travel ahah Thanks @sadortun and also @mtrezza for pushing us to cross the finish line.

Regarding the documentation i'll take a look this week, today it's a public holiday in France :)

Have a nice day guys ! Let's continue on this way 👍 💯

[parseplatformorg]

🎉 This change has been released in version 5.0.0-beta.1

[parseplatformorg]

🎉 This change has been released in version 5.0.0

[vince1995]

Really cool feature. But something's wrong in the docs. There should be a https://parseplatform.org/parse-server/api/5.0.0/SchemaOptions.html site which doesn't exist.

[mtrezza]

See #7805

[lenny-sb]

Is there any documentation regarding this feature?

[mtrezza]

See parse-community/docs#863, to be merged soon

[Moumouls]

Docs are available now: https://docs.parseplatform.org/defined-schema/guide/