At the moment there is support for s3 connections. We would like to extend this to Azure data lake storage and Google cloud storage.
Modifications that have to happen:
flowfile_core/flowfile_core/flowfile/flow_data_engine/cloud_storage_reader.py: The parsing of configs is handled here
flowfile_core/flowfile_core/flowfile/flow_data_engine/flow_data_engine.py: Depending on if the current set-up handles also scenario's for adls and gcs there might be some refactoring needed to it.
- In the CI test template add Azurite and GCS emulator for testing purposes.
At the moment there is support for s3 connections. We would like to extend this to Azure data lake storage and Google cloud storage.
Modifications that have to happen:
flowfile_core/flowfile_core/flowfile/flow_data_engine/cloud_storage_reader.py: The parsing of configs is handled hereflowfile_core/flowfile_core/flowfile/flow_data_engine/flow_data_engine.py: Depending on if the current set-up handles also scenario's for adls and gcs there might be some refactoring needed to it.