Pekko Connector Fault Tolerance Processing

[Susmit07]

Hi Team,

I tested this functionality

• Prepare 2000 files in a test location local inside the pod
• Deploy the app on Kubernetes with multiple pods using the Cluster Singleton setup. Ensure that only one pod is processing files at a time. (which we achieved)
• While the first pod is processing files, kill the pod to simulate failure.
• Monitor the logs of the next pod that takes over. It should resume processing from the last processed file.

But actually its processing from the first one again, in other words no offset is maintained.

Does pekko out of the box supports exactly once or we need to maintain the state in a PersistentVolumes / DB like a key value store along with Pekko Persistence module

Awaiting response

[Susmit07]

@pjfanning @raboof

[pjfanning]

Please don't tag me on these - I'm a volunteer - and I much prefer to see people read docs and ask questions about the docs than just asking questions with no research. We write docs because it is more sustainable than answering everyone's questions.

One option is Pekko Persistence - there are 4 supported types, all can be found in our docs. https://pekko.apache.org

You can also write your own code that persists the offset position.

[Susmit07]

Yeah I do second your opinion, but thanks for the reply, I feel lot of setup/dependencies needs to be done at an application layer, every pull, the flow should check if file processed or not from a distributed store

frameworks like Flink for instance does it out of the box, just trying to connect the dots anyways both has different use case

[raboof]

As discussed in #814 before, you'll have to decide for yourself how you want to achieve exactly-once processing for your system. Indeed you'd have to keep state somewhere - either using Pekko Persistence (e.g. https://pekko.apache.org/docs/pekko/current/typed/index-persistence-durable-state.html). But as you're already working with HDFS and S3, perhaps you could store something or derive where to continue from there?

[Susmit07]

Yes I am going to take s3 route, will have a marker file per directory level, which will have all files metadata IN_PROGRESS, COMPLETED, FAILED

[Susmit07]

Thank you guys, you people are the hope for developer still adapting or new to this framework ..
Keep it up