tiles-daily.csv contains null values for PROGRAM #2251
Assignees
Labels
Comments
|
@sbailey I seem to recall discussing regeneration of the tiles file after Jura was completed. Is it still possible to do that? |
|
@sbailey, @akremin, Please look in The files were produced with this notebook. One key point here: for any tile, frame or exposure exists in both daily and jura, every column in daily is replaced with the corresponding column in jura. This is not mandatory: a subset of columns or rows could be selected, but I would need guidance on which rows or columns to replace and which to leave alone. |
|
Thank you, Ben. This looks very nice. On that key point you made: I think we need to avoid overwriting entries that are specific to the version of the pipeline used to process the data. Examples are the I think the final choice of columns will require Stephen-level sign off, but to make progress while he's away I would recommend the following as a complete list of columns that CAN be altered: tiles file columns to sync: exposures file columns to sync: All of these columns are "facts" (ideally metadata) about the observations or derived from ETC or GFA measurements. Note for tiles I've even removed NEXPS and EXPTIME because those depend on the specific data included in the specprod (we may have flagged expoures in Jura that weren't flagged in daily, and we want to correctly reflect the number of exposures and exposure time in daily). |
|
@akremin, thank you, I'll run some tests on those columns. I also need to test whether there are missing data in the |
|
In addition, @sbailey and I had some discussion on Slack last week. Here are the key points:
|
|
It turns out that even after patching and applying the night cuts above, there are a few tiles that still have missing values, and I'm trying to figure out what to do with these. There are also cases where the tile was successfully patched but some values in the exposures table were not patched, presumably because some exposures are in daily but not jura, for the same tile. The answer may come down to how much work would it be to further try to patch these cases, versus just recreating @akremin do you have any insight? I only got a summary of your discussion with @sbailey, but I believe we're close to a point where further patching and recreating from scratch would require equivalent work, and recreating from scratch would get us into a good place for the long term. |
|
@sbailey, as discussed today, here are the tiles that remain unpatched even after patching with Jura: A few of them have small but still non-zero values of |
|
While investigating the exposures that have And note that these also have |
|
Here is the comprehensive list of tiles that have problematic exposures, including both tiles that have some exposures with It seems like a long list, but it's not actually that long given that there are 14723 daily tiles capable of being loaded right now (14739 - 16 unpatched tiles listed above). |
|
Thinking about this more: the good news is that we can recover missing MJD values from However, the other problem of no valid exposures doesn't have an obvious workaround at this point. |
|
A new problem: for some exposures that are in both |
|
could you provide an example? |
|
|
|
|
thanks! |
|
for what is worth, the gfa catalogs have those are low-quality exposures, so it could make sense that the gfa pipeline failed for those for some reason: |
|
Progress on this issue:
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
The first ~25% of rows in
tiles-daily.csvhave a null value for the PROGRAM column. In contrast specprods such asironhave no such null values.The difference is that
tiles-daily.csvis updated incrementally, whiletiles-iron.csvwould have been generated from scratch.The most likely explanation is that sometime in the past
desi_tsnr_afterburnerhad an issue with identifying the correct value for PROGRAM and left it blank. After the problem was fixed, the file was not regenerated, but instead newer tiles with valid PROGRAM were simply appended to the file.The text was updated successfully, but these errors were encountered: