CUDA takes a very long time to load

[topolarity]

On 1.9 beta:

julia> @time using CUDA
  4.206624 seconds (12.11 M allocations: 891.277 MiB, 5.68% gc time, 1.14% compilation time)

On master:

julia> @time using CUDA
 27.029670 seconds (88.74 M allocations: 10.128 GiB, 5.24% gc time, 0.47% compilation time)

[ViralBShah]

cc @maleadt

[staticfloat]

Bisect blames 96acd4d

[maleadt]

cc @N5N3

[N5N3]

The regression should be caused by the slow path added for correctness in #48441.
I tried to set the threshold of inner Lunion from 3 to 0 (thus we would always skip slow path) and the time cost drecreased dramaticly.