OSTEP behavior on r2 in Aug / Sep 2024

[schlafly]

We've twice seen jumps in the bias level of r2 in darks in the last two weeks.

These are visible in the nightly darks on 20240909 and 20240824 (thanks @LNapolitano , @araichoor ).
https://data.desi.lbl.gov/desi/spectro/redux/daily/nightqa/20240909/dark-20240909.pdf
https://data.desi.lbl.gov/desi/spectro/redux/daily/nightqa/20240824/dark-20240824.pdf

These are ephemeral and don't persist into the morning darks or obviously impact the sframes of science data, but one presumes that sometimes the bias is also fluctuating in a similar way in science data.

Someone should review the OSTEP found for these exposures and for all of the cameras to see to what extent this is an increasing r2 problem vs. just normal. I think we dealt with this a lot back in ~2021 but hadn't seen it much since, and I don't remember what we may have changed to get rid of it, but I may be misremembering.

[sbailey]

A bit of additional information: for the case in the plot above, the OSTEP is a little larger for amp C than the others, but note huge:

$> cd /global/cfs/cdirs/desi/spectro/redux/daily/preproc
$> fitsheader -e 0 20240909/00252286/preproc-r2-00252286.fits.gz  | grep OSTEP
OSTEPA  =   0.6955534458902548 / ADUs (max-min of median overscan per row)      
OSTEPB  =    0.772791065288402 / ADUs (max-min of median overscan per row)      
OSTEPC  =   1.0651512971526245 / ADUs (max-min of median overscan per row)      
OSTEPD  =   0.7674307892739307 / ADUs (max-min of median overscan per row)      

Will need to dig more in the raw data and look at the overscan and see how big that step actually is.

[sybenzvi]

FYI, I extracted the OSTEP data from the preproc headers for all exposures in 202409, plotted the values per amp as a function of date, and put 3 sigma outliers into tables. I dumped all the information into a webpage to make it more tractable:

https://data.desi.lbl.gov/desi/users/sybenzvi/ostep/ostep.html

I did not go systematically through all of the exposures to identify any patterns. I did check all of the CALIB Dark 5 min exposures with outliers (there were 8 in total). None of those exhibited a visible step-like jump in the bias except expid 252286, IDed by Lucas.

However, clicking around a bit, one can find visible jumps in other exposures. For example, r2 - exp 252024, 20290907.

My subjective comment is that the OSTEP in exp 252286 (20240909) does not look remarkable. It's not even the largest OSTEP for r2c from the past 3 weeks (see here).

[julienguy]

This is very likely an issue with the ADC in the readout electronics. We will see how to fix the hardware.

[julienguy]

We should develop a software fix anyway for the time this CCD has this bad behavior. The offset of ~1 ADU is too small to pass the 'OSTEP' threshold to switch to a subtraction of overscan per row. We have to develop another diagnosis for this. See examples of steps in the profiles of the overscan bands covering the left amplifiers (A&C of r2) (2 images for 2 different exposures).