Properly scale MDM profile delivery for 60k+ hosts

[getvictor]

Currently, MDM profile delivery is effectively broken for 60k hosts. See recent loadtest for specific details.

For example, delivering 10 profiles to 60k hosts would result in (10+1)*60k=660k MDM server requests (with multiple DB requests each) over a short period of time.

We can work around this issue by doubling (or tripling) the number of server containers. However, that would be provisioning for the worst case, and those resources would be unused most of the time.

I recommend we explore throttling profile delivery. For example, deliver profiles to 10k hosts every 30 seconds.

Related issue: #23976

[gillespi314]

I recommend we explore throttling profile delivery. For example, deliver profiles to 10k hosts every 30 seconds.

Are you thinking the MDM-protocol endpoints (which is the point where the profiles are technically delivered to the device) would be throttled? Or are you suggesting we implement batching by host uuid somewhere in the Fleet service methods that handle things like writing to the nano queue, dispatching pushes via APNS, etc.?

[lukeheath]

Thanks for filing. It's going to be important for us to get ahead of these scaling concerns before some of our larger rollouts planned in the first half of next year.

I'm prioritizing this to the drafting board for estimation to explore what this would look like. I'm also adding it to Improve MDM architecture story, which I'll review with @georgekarrv at our next 1-1 to discuss how to prioritize.

[github-actions]

This issue is stale because it has been open for 365 days with no activity. Please update the issue if it is still relevant.

[lukeheath]

@getvictor @gillespi314 @JordanMontgomery I agree, we need to increase the number of MDM hosts we're load testing against. I believe it's currently 20K? 60K sounds right in order to make sure we're load testing more hosts than anyone else is using. Any other improvements in fidelity we can make are also important.

I have no idea what that looks likle, or what's needed to achieve it. But I do know it's not something we can defer any longer, as real-world host usage is exceeding our load tests. It is up to @georgekarrv and the MDM group to decide what that is, but we definitely need to get moving on something.

@georgekarrv I'm assigning to you. Let's discuss and work on getting this through drafting in January.