Handle timeouts possible in Khepri minority in rabbit_db_binding

[the-mikedavis]

These changes are split off of #10915 - I'd like to approach these changes incrementally and work through each module so that the diff is readable. This PR covers rabbit_db_binding: create/2 and delete/2 may return {error, timeout} when Khepri is in a minority and this PR updates the direct and indirect callers so that the error is handled gracefully.

[michaelklishin]

I don't object to this change but throwing on the definition import path will halt the entire process. Perhaps we already do that in other places :( It's a good question whether we should continue or halt, "some definitions" can be better than none.

But for cases where the definitions are imported from an external tool on boot (and that happens to early), failing makes more sense.

[the-mikedavis]

I was originally thinking we should refactor rabbit_definitions to import all definitions in one big transaction. Then the side effects like starting vhost or queue processes would happen after the transaction succeeded. It would be a really really big refactor though and definitions import is idempotent so if an import fails partially all you should need to do is re-run it

[the-mikedavis]

We do end up catching these throws here

rabbitmq-server/deps/rabbit/src/rabbit_definitions.erl

Lines 663 to 671 in ff5a324

	_ = try
	WorkPoolFun(M)
	catch {error, E} -> gatherer:in(Gatherer, {error, E});
	_:E:Stacktrace ->
	rabbit_log:debug("Definition import: a work pool operation has thrown an exception ~st, stacktrace: ~p",
	[E, Stacktrace]),
	gatherer:in(Gatherer, {error, E})
	end,
	gatherer:finish(Gatherer)

and convert it into an {error, _} but it will halt the import after the first concurrent_for_all/4 throws. It won't be a crash+stacktrace in the logs at least 😅

[dcorbacho]

If the metadata store is timing out, most likely it won't be the only failing operation. I wonder if this is one of the cases where we should retry the operation. Like if it fails during boot, retry a few times. Or just abort the boot process?

[the-mikedavis]

I'll add a bullet for looking into startup specifically to my todo list. It looks like currently we crash when trying to register projections if you start in a minority during rabbit_khepri:setup/0 which comes before definitions import. (Start a 3-node cluster, enable khepri_db, stop the cluster, start one node.) I think we should have retries there so you have some time to form a majority before we error out and fail the boot. There are also hard matches on ok for when we insert default data

rabbitmq-server/deps/rabbit/src/rabbit.erl

Lines 1139 to 1168 in 2679649

	insert_default_data() ->
	DefaultUser = get_default_data_param(default_user),
	DefaultPass = get_default_data_param(default_pass),
	{ok, DefaultTags} = application:get_env(default_user_tags),
	DefaultVHost = get_default_data_param(default_vhost),
	{ok, [DefaultConfigurePerm, DefaultWritePerm, DefaultReadPerm]} =
	application:get_env(default_permissions),
	DefaultUserBin = rabbit_data_coercion:to_binary(DefaultUser),
	DefaultPassBin = rabbit_data_coercion:to_binary(DefaultPass),
	DefaultVHostBin = rabbit_data_coercion:to_binary(DefaultVHost),
	DefaultConfigurePermBin = rabbit_data_coercion:to_binary(DefaultConfigurePerm),
	DefaultWritePermBin = rabbit_data_coercion:to_binary(DefaultWritePerm),
	DefaultReadPermBin = rabbit_data_coercion:to_binary(DefaultReadPerm),
	ok = rabbit_vhost:add(DefaultVHostBin, <<"Default virtual host">>, [], ?INTERNAL_USER),
	ok = rabbit_auth_backend_internal:add_user(
	DefaultUserBin,
	DefaultPassBin,
	?INTERNAL_USER
	),
	ok = rabbit_auth_backend_internal:set_tags(DefaultUserBin, DefaultTags,
	?INTERNAL_USER),
	ok = rabbit_auth_backend_internal:set_permissions(DefaultUserBin,
	DefaultVHostBin,
	DefaultConfigurePermBin,
	DefaultWritePermBin,
	DefaultReadPermBin,
	?INTERNAL_USER),
	ok.

so I think keeping definitions imports and default data behaving the same is probably ok for now. These operations happen right after we start up Khepri and send commands like register_projections so if we get to this part of the boot we have already been in a majority and probably still are, unless some of the nodes disappear right after boot.

[dcorbacho]

I thought we added retries at some point to wait for other nodes, as we do for Mnesia. I'll have a look

[dcorbacho]

We just wait for the leader, so I opened a new PR waiting for the projections: #11741

[the-mikedavis]

Force-push was a rebase