[BUG] Haversine RBC pytest failing w/ 1 mismatched element in CI

[cjnolet]

During the 21.10 release, there is a pytest failing (the CUDA version seems intermittent at the moment) w/ a single mismatched element.

https://gpuci.gpuopenanalytics.com/job/rapidsai/job/nightly-tests/job/docker-test-cuml/1120/CUDA_VER=11.2,DOCKER_REPO=rapidsai%2Frapidsai-core-dev-nightly,LINUX_VER=centos7,PYTHON_VER=3.7,RAPIDS_VER=21.10/testReport/cuml.test/test_nearest_neighbors/test_nearest_neighbors_rbc_1000_2_haversine_/

[github-actions]

This issue has been labeled inactive-30d due to no recent activity in the past 30 days. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed. This issue will be labeled inactive-90d if there is no activity in the next 60 days.

[levsnv]

Seen this test fail with 139 elements (including the one permitted by the test):
https://gpuci.gpuopenanalytics.com/job/rapidsai/job/gpuci/job/cuml/job/prb/job/cuml-gpu-test/CUDA=11.2,GPU_LABEL=driver-495,LINUX_VER=ubuntu18.04,PYTHON=3.8/1719/
https://gpuci.gpuopenanalytics.com/job/rapidsai/job/gpuci/job/cuml/job/prb/job/cuml-gpu-test/CUDA=11.5,GPU_LABEL=driver-495,LINUX_VER=ubuntu20.04,PYTHON=3.8/1719/

Test Result (2 failures / +2)
cuml.test.test_nearest_neighbors.test_nearest_neighbors_rbc[10000-4-euclidean]
cuml.test.test_nearest_neighbors.test_nearest_neighbors_rbc[10000-25-euclidean]
--
distance = 'euclidean', n_neighbors = 4, nrows = 10000

    @pytest.mark.parametrize('distance', ["euclidean", "haversine"])
    @pytest.mark.parametrize('n_neighbors', [4, 25])
    @pytest.mark.parametrize('nrows', [unit_param(10000), stress_param(70000)])
    def test_nearest_neighbors_rbc(distance, n_neighbors, nrows):
        X, y = make_blobs(n_samples=nrows,
                          centers=25,
                          shuffle=True,
                          n_features=2,
                          cluster_std=3.0,
                          random_state=42)
    
        knn_cu = cuKNN(metric=distance, algorithm="rbc")
        knn_cu.fit(X)
    
        query_rows = int(nrows/2)
    
        rbc_d, rbc_i = knn_cu.kneighbors(X[:query_rows, :],
                                         n_neighbors=n_neighbors)
    
        if distance == 'euclidean':
            # Need to use unexpanded euclidean distance
            pw_dists = cuPW(X, metric="l2")
            brute_i = cp.argsort(pw_dists, axis=1)[:query_rows, :n_neighbors]
            brute_d = cp.sort(pw_dists, axis=1)[:query_rows, :n_neighbors]
        else:
            knn_cu_brute = cuKNN(metric=distance, algorithm="brute")
            knn_cu_brute.fit(X)
    
            brute_d, brute_i = knn_cu_brute.kneighbors(
                X[:query_rows, :], n_neighbors=n_neighbors)
    
        rbc_i = cp.sort(rbc_i, axis=1)
        brute_i = cp.sort(brute_i, axis=1)
    
        # TODO: These are failing with 1 or 2 mismatched elements
        # for very small values of k:
        # https://github.com/rapidsai/cuml/issues/4262
>       assert len(brute_d[brute_d != rbc_d]) <= 1
E       assert 139 <= 1
E        +  where 139 = len(array([0.26635543, 0.14040843, 0.15865709, 0.224747  , 0.22284727,\n       0.31209642, 0.196584  , 0.4371513 , 0.459564... 0.09977522, 0.20649612, 0.20396307, 0.18100378,\n       0.36238492, 0.41610897, 0.14166015, 0.17231807], dtype=float32))

cuml/test/test_nearest_neighbors.py:556: AssertionError

[github-actions]

This issue has been labeled inactive-30d due to no recent activity in the past 30 days. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed. This issue will be labeled inactive-90d if there is no activity in the next 60 days.