Skip to content

[Discussions-4763]Improve export/import csv tool;#4774

Closed
limeng32 wants to merge 2 commits intoapache:masterfrom
limeng32:master
Closed

[Discussions-4763]Improve export/import csv tool;#4774
limeng32 wants to merge 2 commits intoapache:masterfrom
limeng32:master

Conversation

@limeng32
Copy link
Contributor

@limeng32 limeng32 commented Jan 11, 2022

Hello, everyone. I recently used iotdb in the production environment and found that the import tool is almost unusable in the real scene. I studied this piece of code and found that the stream was not used correctly, so I refactored it, and I also added some other functions. What I do is as follows:

  1. Optimize the implementation of import, so that the records in the backup file can be read into the memory, written to the database, and moved out of the memory line by line. The new tool no longer has any requirements on the size of the backup file;
  2. Added compression options. It is a common behavior to compress while exporting. At the same time, importing from compressed files should also be supported. At present, I have implemented it in two ways: gzip and snappy;
  3. There is corresponding progress feedback when importing/exporting. At present, I use feedback every time I export 10 million points or import 1 million points, so that no matter what type of data, I can have a suitable feedback rhythm;
  4. Because the original code is almost completely unused, the new tools are named export-csv-new and import-csv-new, which coexist with existing tools;

@limeng32 limeng32 changed the title Improve export/import csv tool; [Discussions-4763]Improve export/import csv tool; Jan 11, 2022
@coveralls
Copy link

coveralls commented Jan 11, 2022

Coverage Status

Coverage increased (+0.003%) to 67.75% when pulling 99b86ef on limeng32:master into ce154b8 on apache:master.

@limeng32 limeng32 deleted the branch apache:master January 30, 2022 09:39
@limeng32 limeng32 closed this Jan 30, 2022
@limeng32 limeng32 deleted the master branch January 30, 2022 09:39
@limeng32 limeng32 restored the master branch January 30, 2022 09:42
@limeng32 limeng32 deleted the master branch January 30, 2022 09:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants