Unable to read rows. KeyError('name') during domain upload

[spankywetfish]

When uploading domains via CSV the error "Unable to read rows. KeyError('name')" is returned.

This can easily be reproduced by downloading the template, and then uploading it straight away without any modifications.

[IAMPi]

I manage to get a CSV file uploaded and processed. The initial "Unable to read rows. KeyError('name')" appears to be related to spaces after the delimiting comma. If you remove those the error goes away.

From there the file will complain about the whois column. The whois is column should actually be named whois_status. Once the column header was renamed then the CSV causes the following error:

ValueError('Cannot assign "'Enabled'": "Domain.whois_status" must be a "WhoisStatus" instance.')

I couldn't figure out why it was causing this error as I had "Enabled" as an option in my whois statuses table. I ended up deleting the column and the file uploaded correctly.

The last outstanding issue I encountered was with the registrar column. Its set to unique in the DB for some reason so it doesn't allow you to upload multiple domains with the same registrar. The quick solution is probably deleting the column from the CSV. I ended up altering the table and removing the UNIQUE parameter.

Hope that helps

[chrismaddalena]

I actually ran into this today with a new csv file during some testing. I'm looking into what the cause might be. It should not be tripped up by name.

@IAMPi Nice catch on the registrar being unique. The intention is for registrar to be unique in a Registrar model. Then the Domain model would get a foreign key for the registrar. I thought it'd be nice to add the field as a text field for now and circle back with some updates for things like seeing all domains for registrar X, etc. In thinking about the future my brain decided the registrar would be unique - it would be unique in a Registrar model, right? - but it shouldn't be unique in the Domain model right now.

I'll fix that as I look at the csv stuff today.

[chrismaddalena]

I was able to reproduce the issue in two ways:

1. The template provided with Shepherd had a space before name in that header. That did not match the name of the model field, so it caused an error. This was also the case with a few other keys.

2. If the data is saved using certain applications/formats that header becomes \ufeffname when read by Django and Python. That's the byte order mark and causes the same error as the space.

I fixed the template.csv file to remove the spaces. Making sure you save the csv without a BOM "fixes" the second potential issue. I went ahead and instructed Shepherd to decode the csv files as utf-8. This may cause issues with certain characters, but since we're dealing with domain names and those can't contain the characters that usually mess with encoding (e.g. accented characters) I think it'll be OK.

As for the whois status, I just forgot to make sure that was converted to a proper WhoisStatus object. That has been fixed as well.

I am calling this fixed, but leaving this open so others can easily see it. Also this may warrant a bigger change. In the long term it would be good for Shepherd to be able to trim any whitespace from the headers and domain data. Right now the csv rows are passed to the model as an OrderDict which is really nice and convenient, but also means users will have to be diligent about any whitespace before or after the strings. This is most important for the headers and for domain_status, whois_status, and health_status as these are foreign keys and require database queries to create the objects.

Will commit the code today.

[g1ldedm1n1on]

FYI DomainChecker outputs Domain instead of Name as the column header. Renamed that column header to Name and the KeyError goes away but is replaced with "Unable to read rows: Error('new-line character seen in unquoted field - do you need to open the file in universal-newline mode?')"