[FEA] Reorganize and improve Python tests

[vyasr]

Note for developers
This is a meta-issue aiming to categorize a wide range of issues. Developers who want to tackle a specific item from the checklist below should create a new issue for just that item, self-assign that issue, and then link it to the checklist above.

Is your feature request related to a problem? Please describe.
There are currently a number of different problems that make it difficult to find, add, or run tests.

• Tests are partially and inconsistently organized by functionality, data types, and parametrizations, so it's not clear which file tests of a specific function might be in. The partial organization by dtype is particularly confusing because it means that in some files we test a single function across many dtypes, whereas in other files we test many functions for a single dtype.
• Test files are too large and contain too many tests.
• There are currently many tests that raise warnings as well as many xfailed tests that actually xpass.
• Tests are currently slow to run because we rely on excessive parametrization and we test many private APIs. Additionally, cuIO tests are especially slow because the corresponding libcudf APIs are difficult to test, so a greater burden is placed on the Python APIs to capture a wider range of issues.

Describe the solution you'd like
These are tasks that we propose to undertake to address the various issues discussed above:

• See Remove xfail condition from now passing tests #7386. Change pytest to run fail on xpasses by specifying the strict parameter. Here are the relevant pytest docs.
• Make pytest fail when it encounters warnings. We can accomplish this by setting in the appropriate config file (probably setup.cfg). (Done in Raise warnings as errors in the test suite #12468)
  • [FEA] Remove FutureWarnings from Python tests #10363
  • Remove other warnings from Python tests (this was done across a large number of PRs)
• Reorganize test files to match the groupings in our API documentation. See also [QST/Discussion] Reorganize pytests #4730
• Reorganize tests into classes that reflect the actual class hierarchies. For example, tests that should be performed for Series and DataFrame objects can go into a test_indexed_frame.py file.
• Reduce excessive parametrization. Many individual tests are actually run hundreds or even thousands of times because we construct a matrix of parameters. In many cases those parametrizations aren't actually testing anything useful, for instance when a test is parametrized by input size. In cases where we're worried about how the underlying libcudf algorithm behaves when block sizes are exceeded, for instance, we should push those tests down to C++. Python tests should favor fewer parameters and in some cases more specialized tests to handle specific edge cases.
• Where appropriate, we should replace parametrization with fixtures. Fixtures will be typically evaluated lazily unlike parameters, which are materialized when the decorated functions are defined, so changing this should improve collection time and reduce the frequency with which we see failures during collection. Additionally, it makes it easier to debug failing tests when breakpoints are placed in the code; we don't want to hit breakpoints during collection. [FEA] Create collection of dataframes for testing #2530 is related to this and may be resolved by addressing this task, but it's not quite as broad.
• Remove tests of private functions/classes. If those functions/classes have corresponding public APIs, we need to make sure that those APIs are tested. If they are, then the tests of private code may be removed, otherwise those tests should be rewritten in terms of the corresponding public APIs.
• Stop using pd._testing ([FEA] Figure out alternative to pandas._testing #6435)
• Reduce runtime of I/O tests #10001
• Implement standard fixtures or use a similar approach to ensure consistent dtype coverage in tests. As part of this, decide what constitutes sufficient dtype coverage

[bdice]

I have started looking into xfail_strict=true in PR #9998. That will require some significant work to analyze all the xfail tests that pass (1956 tests failed with xfail_strict=true, and most of those are xpassed strict failures). I am going to prioritize solving locally failing tests like #7314 and reduce the number of warnings in the pytest log first, before going back to #9998.

[github-actions]

This issue has been labeled inactive-30d due to no recent activity in the past 30 days. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed. This issue will be labeled inactive-90d if there is no activity in the next 60 days.