Skip to content

[IOTDB-4121] logical optimizations for ImportCsv #6998

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
HTHou merged 3 commits into from
Aug 17, 2022
Merged

[IOTDB-4121] logical optimizations for ImportCsv #6998

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
5319df0
IOTDB-4121 ImportCsv
2b3c511 Aug 15, 2022
5eda2db
IOTDB-4121 ImportCsv :update the user guides
2b3c511 Aug 15, 2022
75985b4
IOTDB-4121 ImportCsv :update the user guides
2b3c511 Aug 16, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 10 additions & 6 deletions cli/src/main/java/org/apache/iotdb/tool/ImportCsv.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -331,10 +331,12 @@ private static void writeDataAlignedByTime(

Set<String> devices = deviceAndMeasurementNames.keySet();
String devicesStr = StringUtils.join(devices, ",");
try {
queryType(devicesStr, headerTypeMap, "Time");
} catch (IoTDBConnectionException e) {
e.printStackTrace();
if (headerTypeMap.isEmpty()) {
try {
queryType(devicesStr, headerTypeMap, "Time");
} catch (IoTDBConnectionException e) {
e.printStackTrace();
}
}

List<String> deviceIds = new ArrayList<>();
Expand Down Expand Up @@ -501,7 +503,9 @@ record -> {
// query the data type in iotdb
if (!typeQueriedDevice.contains(deviceName.get())) {
try {
hasResult = queryType(deviceName.get(), headerTypeMap, "Device");
if (headerTypeMap.isEmpty()) {
hasResult = queryType(deviceName.get(), headerTypeMap, "Device");
}
typeQueriedDevice.add(deviceName.get());
} catch (IoTDBConnectionException e) {
e.printStackTrace();
Expand Down Expand Up @@ -829,7 +833,7 @@ private static Object typeTrans(String value, TSDataType type) {
if (value.startsWith("\"") && value.endsWith("\"")) {
return value.substring(1, value.length() - 1);
}
return null;
return value;
Copy link
Contributor

@HTHou HTHou Aug 15, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Update the user guides in https://github.com/apache/iotdb/blob/master/docs/UserGuide/Write-And-Delete-Data/CSV-Tool.md and https://github.com/apache/iotdb/blob/master/docs/zh/UserGuide/Write-And-Delete-Data/CSV-Tool.md .

case BOOLEAN:
if (!"true".equals(value) && !"false".equals(value)) {
return null;
Expand Down
12 changes: 7 additions & 5 deletions docs/UserGuide/Write-And-Delete-Data/CSV-Tool.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -142,12 +142,14 @@ Time,root.test.t1.str,root.test.t2.str,root.test.t2.int
1970-01-01T08:00:00.002+08:00,"123",,
```

The data aligned by time, and headers with data type.
The data aligned by time, and headers with data type.（Text type data supports double quotation marks and no double quotation marks）

```sql
Time,root.test.t1.str(TEXT),root.test.t2.str(TEXT),root.test.t2.int(INT32)
1970-01-01T08:00:00.001+08:00,"123hello world","123\,abc",100
1970-01-01T08:00:00.002+08:00,"123",,
1970-01-01T08:00:00.002+08:00,123,hello world,123
1970-01-01T08:00:00.003+08:00,"123",,
1970-01-01T08:00:00.004+08:00,123,,12
```

The data aligned by device, and headers without data type.
Expand All @@ -159,13 +161,13 @@ Time,Device,str,int
1970-01-01T08:00:00.001+08:00,root.test.t2,"123\,abc",100
```

The data aligned by device, and headers with data type.
The data aligned by device, and headers with data type.(Text type data supports double quotation marks and no double quotation marks)

```sql
Time,Device,str(TEXT),int(INT32)
1970-01-01T08:00:00.001+08:00,root.test.t1,"123hello world",
1970-01-01T08:00:00.002+08:00,root.test.t1,"123",
1970-01-01T08:00:00.001+08:00,root.test.t2,"123\,abc",100
1970-01-01T08:00:00.002+08:00,root.test.t1,hello world,123
1970-01-01T08:00:00.003+08:00,root.test.t1,,123
```

### Syntax
Expand Down
9 changes: 6 additions & 3 deletions docs/zh/UserGuide/Write-And-Delete-Data/CSV-Tool.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -142,12 +142,14 @@ Time,root.test.t1.str,root.test.t2.str,root.test.t2.int
1970-01-01T08:00:00.002+08:00,"123",,
```

通过时间对齐，并且header中包含数据类型的数据。
通过时间对齐，并且header中包含数据类型的数据。（Text类型数据支持加双引号和不加双引号）

```sql
Time,root.test.t1.str(TEXT),root.test.t2.str(TEXT),root.test.t2.int(INT32)
1970-01-01T08:00:00.001+08:00,"123hello world","123\,abc",100
1970-01-01T08:00:00.002+08:00,"123",,
1970-01-01T08:00:00.002+08:00,123,hello world,123
1970-01-01T08:00:00.003+08:00,"123",,
1970-01-01T08:00:00.004+08:00,123,,12
```

通过设备对齐，并且header中不包含数据类型的数据。
Expand All @@ -159,13 +161,14 @@ Time,Device,str,int
1970-01-01T08:00:00.001+08:00,root.test.t2,"123\,abc",100
```

通过设备对齐，并且header中包含数据类型的数据。
通过设备对齐，并且header中包含数据类型的数据。（Text类型数据支持加双引号和不加双引号）

```sql
Time,Device,str(TEXT),int(INT32)
1970-01-01T08:00:00.001+08:00,root.test.t1,"123hello world",
1970-01-01T08:00:00.002+08:00,root.test.t1,"123",
1970-01-01T08:00:00.001+08:00,root.test.t2,"123\,abc",100
1970-01-01T08:00:00.002+08:00,root.test.t1,hello world,123
```

### 运行方法
Expand Down