fix build failures and also update to scikit-learn OneHotEncoder sparse parameter to sparse_output for newer scikit-learn versions

[imatiach-msft]

fix build failures and also update to scikit-learn OneHotEncoder sparse parameter to sparse_output for newer scikit-learn versions

Contains several updates to latest azureml automl package, updates to use python 3.9 as default env in tests, and various updates to newer packages to resolve test failures.

Copilot summary:

This pull request includes various updates to the CI workflows, dependencies, and test configurations to improve compatibility and functionality. The most important changes focus on updating Python and package versions, modifying CI workflows, and addressing compatibility issues in the codebase.

CI Workflow Updates:

• .github/workflows/CI-python-AutoML.yml: Limited operatingSystem to ubuntu-latest and updated pythonVersion to 3.9. Updated numpy and pytorch installation commands. Removed unnecessary dependencies installation steps. [1] [2]
• .github/workflows/CI-python.yml: Updated pythonVersion to include only 3.9 and 3.10. Added installation steps for backwards-compatible keras for transformers. [1] [2]
• .github/workflows/CI-python-minimal.yml: Added a step to install libomp on MacOS using Homebrew.

Dependency Updates:

• requirements-dev.txt and requirements-automl.txt: Updated transformers version constraint and removed azureml-automl-core. [1] [2] [3] [4]
• python/setup.py: Added packaging and restricted numpy to versions below 2.0.0.

Code Compatibility:

• python/ml_wrappers/dataset/dataset_wrapper.py: Added version check for sklearn to handle changes in OneHotEncoder parameters.
• tests/automl/test_automl_image_model_wrapper.py and tests/automl/test_automl_image_object_detection_model_wrapper.py: Updated Python version constraints and package versions for compatibility with azureml-automl-dnn-vision. [1] [2] [3] [4]

Test Configuration:

• tests/main/test_model_wrapper.py and tests/main/test_tf_model_wrapper.py: Skipped tests for Keras API due to compatibility issues with the latest TensorFlow. [1] [2]
• tests/main/test_text_model_wrapper.py: Skipped test for transformers model due to changes in text pair support.

Miscellaneous:

• Updated URLs in tests/common_vision_utils.py for downloading datasets. [1] [2] [3]

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 33.33333% with 4 lines in your changes missing coverage. Please review.

Project coverage is 33.20%. Comparing base (06df9af) to head (f323622).

Files	Patch %	Lines
python/ml_wrappers/dataset/dataset_wrapper.py	33.33%	4 Missing ⚠️

❗ There is a different number of reports uploaded between BASE (06df9af) and HEAD (f323622). Click for more details.

HEAD has 16 uploads less than BASE

Flag	BASE (06df9af)	HEAD (f323622)
unittests	17	1

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #175      +/-   ##
==========================================
- Coverage   41.66%   33.20%   -8.46%     
==========================================
  Files          27       26       -1     
  Lines        1884     1864      -20     
==========================================
- Hits          785      619     -166     
- Misses       1099     1245     +146     

Flag	Coverage Δ
unittests	33.20% <33.33%> (-8.46%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.