Transition the Database Entry Event handling from experimental to production

[edwbuck]

After a sufficient amount of time has passed, we should remove the experimental categorization of the database entry event performance improvements.

Much of the testing of the experimental feature has already been performed in production environments. This issue is to coordinate any remaining required efforts and the final code changes necessary to effect the change.

[faisal-memon]

Bugs that need to be fixed:

1. Fix event being dropped on error #5030
2. Emit events when Node Selectors are set #5037

[edwbuck]

@evan2645 Please retarget this for 1.11.0. The issues at Uber are just too close to the 1.10.0 release to have enough observation period to build confidence that no new issues are discovered / introduced.

[stevend-uber]

Can we add an exit criteria for this feature to be production ready to have a holistic integration test for all of the different workflows we've talked about in the past few months?

I think this workflow that @faisal-memon is working on faisal-memon#344, is both valuable but complex and we should have a deep integration test for the whole event-based feature to give confidence for other users.

[edwbuck]

@stevend-uber The only bug that we have encountered that is a race condition in the relational database, one that is well known in database access programming and has a known solution.

The solution you mention "is complex" is, in simple terms:

• We rely on the database's guaranteed to issue ID numbers within the events table.
• We don't rely on the database issuing them in order (here is where the prior mistake was made).
• To avoid relying on them being issued in order, we now take note of the skipped IDs and prior to processing new events prior to pulling a new batch.

For handling the request of a holistic integration test, I would like to note that the unit testing can capture detecting skipped ids, and note that unit testing can check that skipped ids are processed in order prior to the currently existing batch processing. This would put the entire solution under test, and is the current test plan.