Exporting the huge data

[code-blooded-guy]

We’re currently having trouble with exporting 7000 records of data from one entity in RC to a CSV. We need to combine this data with another entity as well, and we have tried writing some logic on the front end which calls an API in a loop in order to make the export happen but the execution is taking too much time.

Unfortunately, we haven't been able to find a solution yet, and we would really appreciate your help.

[tejash-jl]

@code-blooded-guy you can create a reporting service which can connect to ES and run complex queries.

[baljeetsingh]

Hi @tejash-jl

here we are trying to join to entities and get a csv report .

Its like for each student record there is details of teacher
Student Name, Student ID, Teacher Name, Teacher Number, Teacher ID, Teacher AttributeX

so for the operations like this where we need to join two entities and create output like above Reporting Service would be the only option?

I guess these kind of joins to get combined output would be common use case people would have used with RC - any built in support for the same on how to achieve this ?

[tejash-jl]

@baljeetsingh yes we are working on it, Sunbird-RC/sunbird-rc-core#205.
but using APIs to return 7000+ records to generate CSV in frontend won't scale I guess,

[parthlawate]

a Report service could be written off Postgress as well as Elastic if the idea is to not use Sunbird RC APIs for it ?

@tejash-jl I would prefer that if it has to be a separate service, it be designed in a way that can be contributed to the core product as exports & reports are very common use cases that I am seeing downstream right now.. Can you advise on the design ?