Stage large numbers of jobs without affecting performance

[epicfaace]

Allow a user to stage as many jobs as they want without affecting performance (goal: 100K)

[epicfaace]

Notes from meeting 7/2/21:

1K workers and 100K staged jobs: what is the shape of a solution that could make sense?

Todo: Make a doc outlining the solution and how that would differ from the current implementation, inefficiencies in current implementation, etc.

For example: maybe doing everything in memory is better than updating the database over and over again

[epicfaace]

Look at Jane's doc, other scheduling solutions

[epicfaace]

@adiprerepa : take a look at bundle_manager.py, Jane's docs to understand what was done in the past.

[epicfaace]

Note: we want to look at timings of different bundle manager stages, Jane looked at timings earlier, see her doc

[adiprerepa]

benchmark on dev