Parse CPS dat files using file representation of column mapping (by parsing SAS script or data dictionary)

[MaxGhenis]

The 700+ lines of hard-coding like below, in each separate CPS loading script...TBH it scares me a bit.

taxdata/cps_data/pycps/cpsmar2013.py

Lines 17 to 19 in 7fa2634

	record["hrecord"] = int(rec[0:1])
	record["h_seq"] = int(rec[1:6])
	record["hhpos"] = int(rec[6:8])

This seems error-prone and tedious, and slows down the process of adding new CPS data.

Another option would be using the ASEC data dictionaries published by Census, e.g. this from 2018. These could be parsed to create dictionaries for each variable corresponding to a start and end position, e.g.

{'HRECORD': [1,1],
 'HSEQ': [2,6]
...
}

then use this dictionary to loop through lines to create records a DataFrame. I'm not a regex expert but I don't think extracting the dict from the data dictionary seems too hard.

[andersonfrailey]

I agree that the hard coding looks scary, but it's all automatically generated by transform_sas.py. It takes one of the SAS files provided by NBER to parse the DAT files (like this one), and pulls the start and end positions for each variable to fill in this template for each year. There are some hard coded parts/tricks in the template to account for including C-TAM data when possible as well.

The process for adding new CPS data is then just to run the new SAS file through transform_sas.py and it'll output cpsmar20xx.py. How this is done can be seen in this part of PR #339.

I'm open to switching from parsing the SAS files to parsing the data dictionaries though. The only reason I wrote this using the SAS scripts is because those are what I first found when I looked at how to parse those DAT files. Any other improvements that come from switching to data dictionaries would be great as well.

[MaxGhenis]

Thanks @andersonfrailey, that's a relief 😄 . I'll change this issue to be about storing the result of that parsing as a file (dictionary seems appropriate to me but could be something else) and then loading that file in the script. I don't think it matters much whether the SAS script or the data dictionary is the source of that parsing, but doing this would avoid code duplication (even automated code duplication) and be generally cleaner IMO.

[andersonfrailey]

@MaxGhenis, I like the dictionary idea you proposed in your first comment. We'll still need separate dictionaries for each year, but it could definitely result in cleaner code. What do you think would be the best way to save those dictionaries? A file full of them or maybe pickled?

I could see us having one "master" dictionary with keys for each year we support that link to three sub dictionaries for each type of record (household, person, family). Then we could just call each one as needed.

[MaxGhenis]

A pickled master dictionary for all years, or a CSV with the same data, SGTM.

[andersonfrailey]

Perfect. I think a pickled dictionary is easiest for me to wrap my head around. I'll get a PR up.