[Spark Load][Bug] Keep the column splitting in spark load consistent with broker load / mini load

[xy720]

Proposed changes

There is a 4 columns source data:

1|1|jim|2|
2|1|grace|2|
3|2|tom|2|
4|3|bush|3|
5|3|helen|3|
5|3|helen|6|
6|3|helen|3|
6|3|helen|3|
...

Given the same column terminator '|', broker load determines that it is 5 columns, and spark load determines that it is 4 columns.

And there is another 4 columns source

1|1|jim|2
2|1|grace|2
3|2|tom|2
4|3|bush|3
5|3|helen|3
5|3|helen|6
6|3|helen|3
6|3|helen|3
...

Given the same column terminator '|', both the broker load and spark load determines that it is 4 columns.

To Reproduce
Steps to reproduce the behavior:

1. Submit a broker load.

load label ssb_db.broker_load_label 
( 
    data infile ("hdfs://ymy-host:port/user/palo/table1") 
    into table test_tbl 
    COLUMNS TERMINATED BY "|" 
    (k1,k2,name,clicks ) 
) 
with broker "doris" ("username"  =  "test", "password"  =  "test");

2. Submit a spark load.

load label ssb_db.spark_load_label 
( data infile ("hdfs://ymy-host:port/user/palo/table1") 
    into table test_tbl 
    COLUMNS TERMINATED BY "|" 
    (k1,k2,name,clicks ) 
) with resource "spark0" 
("spark.executor.memory"  =  "24g", "spark.executor.cores"  =  "2", "spark.executor.instances"  =  "8");

3. Broker Load will report an error "quality not good enough to cancel"

The reson of this bug
This is because the first character and the last character of a line are not considered to be delimeter in spark load.

Types of changes

What types of changes does your code introduce to Doris?
Put an x in the boxes that apply

• Bugfix (non-breaking change which fixes an issue)

Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code.

• I have create an issue on (Fix [Spark Load][Bug] The number of columns in broker load and spark load is different #4520 ), and have described the bug/feature there in detail

[morningman]

if line is an empty string, this method should return an empty string array.
But here you will return a string array with one empty string in it.

[xy720]

fixed it

[morningman]

LGTM

[PasunuriSrinidhi]

import csv

with open('input_file.csv') as f:
reader = csv.reader(f, delimiter='|')
for row in reader:
# Handle case where first column has extra character
if row[0][0] != '1':
row[0] = row[0][1:]
# Handle case where last column has extra character
if row[-1][-1] != '3':
row[-1] = row[-1][:-1]
# Process the row as usual
# ...

In this example, I have used the csv module to read the input data and split it into columns using the | delimiter. Then I will check the first and last columns of each row for extra characters and remove them if necessary.