azureml-dataprep-rslex doesn't support custom CA bundles

[tomasvanpottelbergh]

• Package Name: azureml-dataprep-rslex
• Package Version: 2.18.4
• Operating System: macOS 13.4.1
• Python Version: 3.10.10

Describe the bug
When using the azureml-fsspec package behind a proxy with TLS inspection I get the following error:
[PyDatastoreSource::list_directory] fails with error: ConnectionFailure { source: Some(ConnectionFailure { message: "Failed to connect to Azure Machine Learning services to resolve Datastore.", source: Some(hyper::Error(Connect, Custom { kind: Other, error: Custom { kind: UnexpectedEof, error: "tls handshake eof" } })) }) }

To Reproduce
Steps to reproduce the behavior:

1. Use a HTTP proxy with TLS inspection (custom CA)
2. Add the CA to the system Keychain
3. Use the azureml-fsspec package

Expected behavior
The package should use the CA bundle available in the system Keychain, or provide a way to override the CA using the SSL_CERT_FILE environment variable for example.

[swathipil]

Hi @tomasvanpottelbergh - Thanks for opening an issue! We'll investigate asap.

@azureml-github - Can you please take a look? Thanks!