Optimize the Import subscribers, instead of using single query used bulk insert / upsert.

[amrishshah]

Optimize the Import subscribers, instead of using single query used bulk insert / upsert. This will help to import subscribers faster compare to single query.

[knadh]

Hi @amrishshah. Have you benchmarked this against the current approach? How big of an improvement is the JSON approach?

The import isn't doing one round trip query per record currently, by the way. It's batching N inserts doing doing a single commit.

[amrishshah]

@knadh , I agree it is taking one round trip, but I think while committing the transaction it will take time as there is a SELECT query while updating the subscriber_lists

I am attaching screenshot of 30k User

This is taking approx. 1 minute to import.

with bulk import it is taking 3 sec.

listmonk=# select count(id) from subscribers;
count

31654
(1 row)

listmonk=# select count(*) from subscriber_lists;
count

63308
(1 row)

[knadh]

Hi @amrishshah. Finally got around to reviewing this. 30k users taking 60 seconds to import is very unusual. From the very first release, I've benchmarked uploads at around ~10k/sec on my commodity laptop. Can you share your system's specs?

I just re-tested it with v5.0.3 and here's 50k records being imported in ~5 seconds on my system.

[amrishshah]

Hi @knadh, my laptop configuration is 16 gb ram and SSD hard disk.

[amrishshah]

hi @knadh , if it is working let not waste time and closing the request.