Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[feature-wip](new-scan) Support stream load with csv in new scan framework #13354

Merged
merged 5 commits into from
Oct 17, 2022
Merged

[feature-wip](new-scan) Support stream load with csv in new scan framework #13354

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
1e0195b
[feature-wip](new-scan) refactor csv reader and add test cases
morningman Oct 12, 2022
2042754
1
morningman Oct 15, 2022
37bf60c
fix bug
morningman Oct 15, 2022
74d36bd
fix bug
morningman Oct 15, 2022
c8e004e
add missing data
morningman Oct 16, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fix bug
  • Loading branch information
@morningman
morningman committed Oct 15, 2022
commit 74d36bd276c369b1b4b0f6fd846c4949a11e82e3
3 changes: 3 additions & 0 deletions be/src/exec/broker_scanner.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -106,6 +106,9 @@ class BrokerScanner : public BaseScanner {
int _line_delimiter_length;

// Reader
// _cur_file_reader_s is for stream load pipe reader,
// and _cur_file_reader is for other file reader.
// TODO: refactor this to use only shared_ptr or unique_ptr
std::unique_ptr<FileReader> _cur_file_reader;
std::shared_ptr<FileReader> _cur_file_reader_s;
FileReader* _real_reader;
Expand Down
3 changes: 3 additions & 0 deletions be/src/exec/json_scanner.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,9 @@ class JsonScanner : public BaseScanner {
int _line_delimiter_length;

// Reader
// _cur_file_reader_s is for stream load pipe reader,
// and _cur_file_reader is for other file reader.
// TODO: refactor this to use only shared_ptr or unique_ptr
std::unique_ptr<FileReader> _cur_file_reader;
std::shared_ptr<FileReader> _cur_file_reader_s;
FileReader* _real_reader;
Expand Down
3 changes: 3 additions & 0 deletions be/src/vec/exec/format/csv/csv_reader.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,9 @@ class CsvReader : public GenericReader {
const TFileRangeDesc& _range;
const std::vector<SlotDescriptor*>& _file_slot_descs;

// _file_reader_s is for stream load pipe reader,
// and _file_reader is for other file reader.
// TODO: refactor this to use only shared_ptr or unique_ptr
std::unique_ptr<FileReader> _file_reader;
std::shared_ptr<FileReader> _file_reader_s;
std::unique_ptr<LineReader> _line_reader;
Expand Down