Use Numba Config to turn on Pynvjitlink Features

[isVoid]

Description

Numba-cuda 0.0.18+ merged in a new feature and made the old way of patching linker with pynvjitlink's patch_numba_linker no longer usable by downstream libraries. The current state of Numba-cuda requires that downstream libraries to enable pynvjitlink features only via CUDA_ENABLE_PYNVJITLINK environment variable. A recent PR NVIDIA/numba-cuda#91 makes it so that the features can be turned on by a config variable at runtime.

This PR is an integration test with that PR and changing the way how pynvjitlink is enabled in cuDF. It enables cuDF to use Numba-cuda since 0.2.0+ (which contains the config change).

Supercedes #17359

Checklist

• I am familiar with the Contributing Guidelines.
• New or existing tests cover these changes.
• The documentation is up to date with these changes.

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[bdice]

We may want to keep an upper bound here. Numba-cuda is not stable yet, afaik.

[pentschev]

There's a suggestion from Bradley, but otherwise LGTM. Thanks @isVoid .

[vyasr]

#17359 also has some changes in aggregation.pyx, do we not need those?

[isVoid]

#17359 also has some changes in aggregation.pyx, do we not need those?

I think those changes are to avoid importing any of cudf.utils.cudautils (which transitively imports numba.cuda) in the cudf startup phase. We don't want that because before NVIDIA/numba-cuda#91, MVC mode is enabled via setting the ENABLE_PYNVJITLINK numba config before importing numba-cuda. With that PR merged, we removed such ordering requirement, aka, we can either import numba-cuda first, then change the config to enable/disable nvjitlinker when needed; Or change the setting, then import numba-cuda.

[isVoid]

The doc-build CI job is failing:

dumping search index in English (code: en)... done
dumping object inventory... done
build finished with problems, 21 warnings (with warnings treated as errors).
make: *** [Makefile:20: dirhtml] Error 1
/__w/cudf/cudf

But I don't see any warnings output in the CI. Are there any warning suppression in the CI?

[vyasr]

But I don't see any warnings output in the CI. Are there any warning suppression in the CI?

No, you probably just need to scroll up further. There are a set of warnings printed after all the "reading sources" lines in before all of the "writing output" lines.

I see them here, but they look odd and completely unrelated to the changes in this PR so I am skeptical of how meaningful they are. I've kicked off a rerun to see if they show up again.

[isVoid]

But I don't see any warnings output in the CI. Are there any warning suppression in the CI?

No, you probably just need to scroll up further. There are a set of warnings printed after all the "reading sources" lines in before all of the "writing output" lines.

I see them here, but they look odd and completely unrelated to the changes in this PR so I am skeptical of how meaningful they are. I've kicked off a rerun to see if they show up again.

This is a related issue: sphinx-doc/sphinx#12589. It suggests there maybe 2 sources of cause for this warning, one is using automodule inside the autosummary directive, the other is using default_role=autolink for sphinx configuration. I'm not sure either is applicable to cuDF though.

[EDIT]: The error seems to have disappeared.

[vyasr]

/merge