[Python][CI] The "Without Pandas" CI build to run nopandas tests actually has pandas installed #36541
Description
See for example https://github.com/ursacomputing/crossbow/actions/runs/5476945590/jobs/9975241254, the pandas tests are not skipped.
This was a consequence of the combination of adding pandas to the docs requirements (#13714) and installing conda_env_sphinx.txt into the base conda-python docker image (https://github.com/apache/arrow/pull/11567/files#diff-63f460890faace8e0ac2e8e679de7ff8e4ff46a6ac512d53a85fa142b714d2c9)