Close scheduler on module exit

[coolreader18]

Description of Changes

Fixes clockworklabs/SpacetimeDBPrivate#1449. Turns out we weren't ever calling Module::close(), and thus were never calling Scheduler::close(). I'm still not able to actually repro the issue but if anything is the cause, this would be it. I also made it so exit().await won't return until the scheduler is fully closed, so that we won't run into race conditions.

Expected complexity level and risk

1

Testing

• If anyone able to repro the issue would be willing to test (or tell me how to repro) that would be super helpful

[joshua-spacetime]

I'm not sure how you would write a non-timing based regression test for this.

You could manually test it by writing a scheduled reducer that updates some counter, inserts into an auto_inc table, etc. Publishing the module with -c a few times should affect the counter's rate of change before your fix. It shouldn't have any effect after your change.

[rekhoff]

Not sure if this is an ideal test, but the follow was performed:

Shared test setup properties:

• Windows 11
• Running from locally built spacetime-cli.exe using respective repos
• Monitored the file-size of \AppData\Local\SpacetimeDB\data\control-db\db.
• Used spacetime server clear between tests to start with a fresh db file.

Test 1 - Baseline:

Branch release/1.0.1, using a single published rust project:

• Published the project default Hello, World! module.
• Did not publish any other modules or interact with the spacetime process during test.
  Results: ~1,430 bytes per minute growth of db file.

Test 2 - Pre-change:

Branch release/1.0.1, using a single published rust project:

• Published a project with:

1. A table_that_will_grow table containing an auto_inc u64 counter and a string.
2. A grow_table_timer scheduled table that called a the grow table reducer.
3. A grow_table reducer that adds a new entry the table_that_will_grow, with a 5554-character long string.
4. The init reducer that schedules the grow_table_timer to run every 50 milliseconds.

• Periodically re-published the same module, using the -c -y flags several times during test.
  Results: ~64,678 bytes per minute growth of db file.

Test 3 - Post-change:

Branch noa/close-scheduler, using the same single published rust project from Test 2:

• Periodically re-published the same module, using the -c -y flags several times during test.
  Results: ~62,695 bytes per minute growth of db file.

If my testing methodology is valid, then it would appear that this PR change had little effect.

[joshua-spacetime]

@rekhoff do you know what the most recent values in the auto_inc column were for both tests?

[rekhoff]

Okay, additional test checking against the replicas directory, using branch noa/close-scheduler, I can confirm:

• Starting from a clean server, publishing the module from Test 2 in my previous comment, the folder would grow ever 50 milliseconds.
• Upon first publishing, a directory named 1 was generated under replicas.
• Upon republishing, a directory named 3 was generated under replicas.
• After 3 was generated, directory 1 no longer grew in size, and 3 was now growing every 50 milliseconds.
  Note: The spacetime-cli was not restarted during republishing, as that would invalidate this test.

Upon testing the same on master branch, the same behavior appears to be true. The newly added folder grows, but the pervious created folder does not.

[rekhoff]

Okay, testing against master and testing against noa/close-scheduler, rebuilding the CLI each time netted the same results.
@joshua-spacetime , the auto_inc column restarted at 1 for each republish as well, and properly incremented by 1 each time the grow_table reducer from the test was called.

[rekhoff]

As an extra data point, I also tested against BitCraft including running bots, and found the same behavior as described previously.

[joshua-spacetime]

Thanks for testing @rekhoff!

I don't believe there was ever a bug with this code, but I'm fine to approve if you think these changes are more explicit @coolreader18.