[Enhancement] Support streaming files from a remote storage Blob, S3, etc

[stuarthrichardson]

Provide the ability to be able to support streaming files from a remote storage AWS, S3, etc. After using the file-pulse connector, I am now having requirements to be able ingest data from files on remote storage as focus is moving to cloud storage. More systems are dumping files into Azure blob storage and AWS S3 buckets, and this trend will only grow.

To support the different favours of remote storage, each supported remote storage I think would need a version of the FSDirectoryWalker and FileInputReader, as the configuration requirements would be different for each. Also I think some of the internal interfaces would need to change too.

I really like how to have implemented the "filter chain", if could be possible to implement something along the lines where there is "fs" configuration for the different storage favours.

I would be happy to contribute.

What do you think?

[fhussonnois]

Hi @stuarthrichardson, thank you for your proposal. You're right, it would be awsome to support remote storage in Kafka Pulse. Actually you are not the first person to ask for that improvment.

But yes, some internal interfaces need to be updated to not use the java File object.

Here are then non-exhaustive list of classes that must be updated:

• FSDirectoryWalker must be updated to return a Collection<URI> instead of a Collection<File> so that we implement one walker per cloud-provider.

• FileContext should be converted into an interface (not sure about that).

• FileSystemScanner

• FileInputIterable

• DefaultFileRecordsPollingConsumer

I would greatly appreciate your help if you want to work on that part ^^

[stuarthrichardson]

Hey @fhussonnois, I am going to familiarize myself with the internal workings of this connector to help...

[fhussonnois]

Hi @stuarthrichardson, hope you're well, just an update to let you known that I push a new branch https://github.com/streamthoughts/kafka-connect-file-pulse/tree/subtask-support-remote-storage which contains the updates required to support remote storage. The FSDirectoryWalker and FileInputReader interfaces has been updated.

[fhussonnois]

Follow-up issues are :

• [Enhancement] add support for AWS S3 [Enhancement] add support for AWS S3 #111
• [Enhancement] add support for Azure Blob Storage [Enhancement] add support for Azure Blob Storage #112