Support multiple databases in the same application with inbox/outbox mechanics

[jeremydmiller]

We can already do this with the Marten + Wolverine combination, but make sure that a Wolverine app can target multiple databases with any combination of EF Core, Marten, and RavenDb

Notes:

• PersistWith(*****) has to be smart enough to be additive based on is it really the same database
• Test the agent assignment w/ different databases

[jeremydmiller]

Notes

• WolverineRuntime will need to keep a list of message stores, make sure that it can sort out duplicates
• Pull the DeadLetterAdminServiceCollection from CW into Wolverine itself, and maybe even merge it into Wolverine
• Test the durability agents to make sure you have one for each unique database
• Fold the ancillary stores? Or just say that the first message store is the master, or that the "main" store in a multi-tenanted store is the main
• Track ancillary stores by marker type too. Memoize in its own ImHashMap. Adjust OutboxedSessionFactory<T>
• Need to look over OutboxedSessionFactory and OutboxedSessionFactory<T> about how it finds Wolverine settings and enables things like the forwarding. Find the message store by Uri somehow instead of just going for IWolverineRuntime.Storage
• Crap, PostgresqlTransport probably needs to be aware of all the databases. Do you allow queueing beyond the "main" store? Named brokers like everything else?
• When you register message stores, require a name?
• MessageDatabaseDiscovery needs to be aware of multiple databases
• The ExternalDbTransportExtensions business needs to be aware of multiple databases
• Allow you to register databases w/ a BrokerName
• StorageCommand needs to be aware
• New extension methods to rebuild the message storage at will from an IHost. Or to clear it out.
• IWolverineRuntime.Storage because the collection?

[jeremydmiller]

More Notes

• The existing PersistWithSqlServer() and PersistWithPostgresql() should throw if there is already a main registration for IMessageStore.
• Additional registrations w/ `PersistWithAncillarySqlServer(*****);
• On startup, if there's no derived known main database, blow up right then and there.
• Maybe add IMessageStore.Role where Role == Main/Ancillary/Tenant
• Do all the validation at start up time
• Add some additional log output for the databases? Add the databases to the Describe() functionality?

[jeremydmiller]

Assumption

• NOT going to try to support postgres or sql server transports against ancillary databases as part of this story or probably 5.0
• NOT going to try to support external tables upfront, or at least leave that for a separate story

Tasks

• Add StorageRole enum with Main/Ancillary/Tenant values
• Add StorageRole Role property to IMessageStore
• Test role against the main database
• Test role against tenant databases added by multi-tenancy w/ Marten
• Test role against tenant databases added by multi-tenancy w/ EF Core
• Test role against ancillary Marten stores
• WolverineRuntime takes in a list of IMessageStore, finds the main, throws if more than one
• Keeps a collection of ancillary stores
• Organize IAncillaryMessageStore by marker type directly on IWolverineRuntime
• Ability to mark some postgres or sql server stores as "ancillary"
• Test that we're building separate durability agents for each ancillary database
• Make StorageCommand aware of all
• New extension methods to rebuild the message storage at will from an IHost. Or to clear it out. Would be helpful
• Build a sample app against multiple EF Core DbContext to different databases
• Review every piece of code that does a downcast on MultiTenantedMessageStore to look for duplication
• Put the new MessageStoreCollection as a public member on IWolverineStore
• Delete IMessageDatabaseSource
• Review all usages of IWolverineRuntime.Storage and use TryFindMainMessageStore<T>() instead as necessary

[jeremydmiller]

Dealing w/ Ancillary Stores

• Register AncillaryMessageStoreApplication<T> as a singleton in the IoC container
• In EF Core transactional middleware, if there's a ancillary store registered for the DbContext, add middleware for AncillaryMessageStoreApplication<T>.Apply
• Ensure that DurableReceiver uses DelegatingMessageInbox if any ancillary stores
• Ensure that DurableSendingAgent uses DelegatingMessageOutbox if any ancillary stores
• E2E tests with EF Core targeting an ancillary database

[jeremydmiller]

Good lord, this one hurt. Going in within an hour though.