Skip to content

Loading big CSV files not safe #95

@garlontas

Description

@garlontas

From __csv_loader.py:

def __load_csv(file_path, cast, delimiter, encoding):
    """Load a CSV file and convert it into a list of namedtuples"""
    # skipcq: PTC-W6004
    with open(file_path, mode='r', newline='', encoding=encoding) as csvfile:
        csvreader = reader(csvfile, delimiter=delimiter)

        # Create a namedtuple type, casting the header values to int or float if possible
        header = __get_csv_header(csvreader)

        Row = namedtuple('Row', list(header))

        mapper = LoaderUtils.try_cast if cast else lambda x: x

        # Process the data, casting values to int or float if possible
        data = [Row(*[mapper(value) for value in row]) for row in csvreader]
    return data

what happens if the file is too big to fit in memory?...

Originally posted by @adrian-herscu in #94 (comment)

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

Projects

Status

🆕 New

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions