Fix flaky test_rebalance

[gjoseph92]

Reduce memory spikes during data transfer with test_rebalance, by using NumPy arrays (zero-copy) and more, smaller keys.

• Closes Regression: P2P shuffle skeleton (#5520) causes test flakiness #5688
• Tests added / passed
• Passes pre-commit run --all-files

[gjoseph92]

@crusaderky this still failed once: https://github.com/dask/distributed/runs/4927021521?check_suite_focus=true#step:12:1626

>       assert 30 <= ndata[a] <= 70
E       assert 80 <= 70

But in stderr for that test, there's no longer anything about unmanaged memory use being high, or the worker pausing or restarting. So is it possible that the rebalance heuristics are just different for 100 small keys vs 10 bigger ones, and the 30-70 numbers should change? I just multiplied your original 3-7 by 10, no idea if those targets still make sense.

[gjoseph92]

Note that with #5695, this may not be necessary. However, the fact that it was so sensitive to a 10mb change in worker memory means it probably should be made less sensitive anyway.

[crusaderky]

Note that with #5695, this may not be necessary. However, the fact that it was so sensitive to a 10mb change in worker memory means it probably should be made less sensitive anyway.

Indeed. I think the whole suite requires a more intensive facelift. See #5697.

[crusaderky]

Superseded by #5697