Skip to content

[YSQL] Support loading large data sets into YSQL using COPY #5241

Closed
@rkarthik007

Description

This is a master issue to track improving transactions which have a very large number of operations, as well as provide various options to batch load data more efficiently.

Status Feature GitHub Issue(s) Comments
Support for large transactions #1923 v2.3.0
Large transactional writes leave large memtables #1906 v2.3.0
Fix OOM when running large COPY FROM #5453 v2.3.1
Support txn batch size for COPY FROM to improve performance for large datasets #2855 v2.3.0
Fix OOM when running large \copy (or COPY FROM stdin) #5603 v2.3.3
Support txn batch size for \copy (and COPY FROM stdin) to improve performance for large datasets #6069 v2.3.3

As noted in the referenced issues above, these were tested (on a small local cluster) with data sets of at least up to 15gb in size (corresponding to ~375 million rows).

Analytics

Metadata

Assignees

Labels

area/docdbYugabyteDB core featuresarea/ysqlYugabyte SQL (YSQL)kind/enhancementThis is an enhancement of an existing featureroadmap-tracking-issueThis issue tracks a major roadmap item, and usually appears in the roadmap list.

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions