Data Import: Check to verify that the imported data matches the data type specified

[RDmitchell]

SHA: b264666

User request

Does SEED check, during data import, that the imported data matches the column data type specified in mapping?

There have been issues with the data type needing to be changed after import because of issues with the data not matching the data type.

Seems like it might be useful if SEED could do some checking at import to hopefully prevent this from happening.

No idea how complicated that might be.

[perryr16]

If a user imports data that does not match the expected data type, SEED will attempt to parse it, and if it fails it will default to None or blank. (This isn't an issue for Text or extra data fields. Extra data is saved in the db as a string, and anything can be converted to a string.)

The user would see this during the "Save Mapping" step. For example, I uploaded a file with invalid data types and the mapping screen shows blanks on the preview.

Those blanks will not overwrite the existing data UNLESS a column has "Recognize Empty" enabled, then it will be overwritten with a blank.

[RDmitchell]

@perryr16 -- I think the user is going to be confused about why their data has not been imported.

Is there a way we can either tell the user why, or possibly let them know in the mapping stage that they have a data type mismatch for Canonical Fields (which users don't understand to start with) ?

In the Mapping Step, if the user hasn't defined a matching field, SEED displays a red error message and won't let the user go to the next step unless they fix it.

Could we do something like that for this case? I think otherwise the users will not know what is going on with their data.

[perryr16]

@RDmitchell

How about a warning like this? Note that on this page we only have access to the first five rows of data, so any values in rows 6 and beyond wouldn't be caught.

Checking all possible mismatches would need to happen on the next screen (Save Mappings), but that could introduce noticeable lag if the file contains several thousand rows.

[RDmitchell]

That looks good -- do we want the color to be red, so that users will definitely see it?

[RDmitchell]

@perryr16 -- is there a way that we can check all the rows? Only checking the first 5 rows might not really be good enough?

@kflemin -- what are your thoughts about this?

[perryr16]

@RDmitchell -- I was leaning yellow since this isn't an 'error', more of a 'warning'.

Checking all possible mismatches would need to happen on the next screen (Save Mappings), but that could introduce noticeable lag if the file contains several thousand rows.

[RDmitchell]

@perryr16 -- You and @kflemin should decide, but this can have pretty major consequences in the imported data, so it might be worth highlighting the message in red.

And if the checking on all the records can happen in the review mappings screen, with the same warning if there are issues, that is fine !!

Thanks!