Skip to content

[Discussions-4763]Restructure export/import csv tool;#4764

Closed
limeng32 wants to merge 1 commit intoapache:rel/0.12from
limeng32:rel/0.12
Closed

[Discussions-4763]Restructure export/import csv tool;#4764
limeng32 wants to merge 1 commit intoapache:rel/0.12from
limeng32:rel/0.12

Conversation

@limeng32
Copy link
Contributor

Hello, everyone. I recently used iotdb in the production environment and found that the import tool is almost unusable in the real scene. I studied this piece of code and found that the stream was not used correctly, so I refactored it, and I also added some other functions. What I do is as follows:

  1. Optimize the implementation of import, so that the records in the backup file can be read into the memory, written to the database, and moved out of the memory line by line. The new tool no longer has any requirements on the size of the backup file;
  2. Added compression options. It is a common behavior to compress while exporting. At the same time, importing from compressed files should also be supported. At present, I have implemented it in two ways: gzip and snappy;
  3. There is corresponding progress feedback when importing/exporting. At present, I use feedback every time I export 10 million points or import 1 million points, so that no matter what type of data, I can have a suitable feedback rhythm;

Because the original code is almost completely unused, the new tools are named export-csv-new and import-csv-new, which coexist with existing tools;

@limeng32 limeng32 changed the title Restructure export/import csv tool; [Discussions-4763]Restructure export/import csv tool; Jan 11, 2022
@limeng32
Copy link
Contributor Author

see #4774

@limeng32 limeng32 closed this Jan 18, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant