[YSQL] Support loading large data sets into YSQL using COPY #5241
Closed
Description
This is a master issue to track improving transactions which have a very large number of operations, as well as provide various options to batch load data more efficiently.
Status | Feature | GitHub Issue(s) | Comments |
---|---|---|---|
✅ | Support for large transactions | #1923 | v2.3.0 |
✅ | Large transactional writes leave large memtables | #1906 | v2.3.0 |
✅ | Fix OOM when running large COPY FROM | #5453 | v2.3.1 |
✅ | Support txn batch size for COPY FROM to improve performance for large datasets | #2855 | v2.3.0 |
✅ | Fix OOM when running large \copy (or COPY FROM stdin) |
#5603 | v2.3.3 |
✅ | Support txn batch size for \copy (and COPY FROM stdin) to improve performance for large datasets |
#6069 | v2.3.3 |
As noted in the referenced issues above, these were tested (on a small local cluster) with data sets of at least up to 15gb
in size (corresponding to ~375
million rows).