Logging Celery Export Duration and Optimizing for Large Submissions

[KipSigei]

Suggested Feature / Enhancement

We need to log the duration of Celery exports and analyze the results to identify potential optimizations, especially for forms with large submissions of over 50,000 submissions. Currently, we're experiencing significant delays with exports taking longer times to complete.

Benefits of implementing the feature/enhancement

Improved performance: By logging the duration of Celery exports and analyzing the results, we can identify potential bottlenecks and areas for optimization. This will allow us to improve the performance of our export process, resulting in faster exports and a better user experience.

Better resource management: Analyzing the results of Celery exports will help us identify any issues with resource usage, such as high CPU or memory usage. This will allow us to better manage our resources and ensure that exports are not causing issues for other processes on the server.

Improved scalability: As the application grows and the number of form submissions continue to increase, it will become increasingly important to optimize the exports. By identifying optimization options, we can ensure that the export process is able to scale with the app and handle large volumes of form submissions.