Measured total sometimes wrong when Crew Chief says hall is OFF

[slominskir]

It appears the logic used to clean up the measured values is not quite right as there are some hours (some rows) of data that do not add up to an hour. See: https://logbooks.jlab.org/entry/4358599

Example rows:

• April 11 DAY Hall A hour 8
• April 11 SWING Hall A hour 16 & 17

For context, the measured values are simply read from EPICS by BTM and are written by apps that are unknown to me. Unfortunately, when the Crew Chief sets Hall OFF in BOOM app (Hall A PV = BOOMHLAOFF_h , the experimenter PVs appear to not recognize that in the back end IOC / whomever is writing to those PVs. So BTM attempts to clean up the numbers, but looks like missed a spot. At quick glance my guess is due to hour being split with partial off. I'm pretty sure BTM currently punts (does nothing) to clean up when partial off detected - because it is difficult to know what is expected in this case (maybe equally remove seconds from other buckets?). See cleanup code here:

btm/src/main/java/org/jlab/btm/business/service/epics/HourRounder.java

Lines 38 to 108 in 5fc3a45

	/**
	* Round an experimenter hall hour time accounting (both experimenter and accelerator statuses) to
	* a whole hour.
	*
	* <p>The status 'off' is a shared status of both experimenter and accelerator statuses so
	* rounding to correct one set of statuses could affect the other set. Therefore, 'off' is
	* truncated to range, then UED/OFF fix applied, and finally during rounding of each mutually
	* exclusive set OFF is then static.
	*
	* @param hour the experimenter hall hour.
	*/
	public void roundExpHour(ExpHour hour) {
	// Make sure shared status Off is within range 0 - 3600
	short[] statuses = new short[1];
	statuses[0] = hour.getOffSeconds();
	truncateToRange(statuses);
	hour.setOffSeconds(statuses[0]);
	// Note: we only adjust for extremes;
	// We only fix scenario when entire hour is erroneously one of these metric OR roughly matches
	// amount of OFF;
	// We don't subtract difference and keep some off and some of these other metrics;
	// There is an order/precedence of cleanup: UED, ER, PCC;
	// Someone should Fix EPICS IOC measure logic!
	// UED
	adjustOffAndExperimentMetric(
	hour,
	new ExperimentHourMetric() {
	@Override
	short getSeconds() {
	return hour.getUedSeconds();
	}
	@Override
	void setSeconds(short seconds) {
	hour.setUedSeconds(seconds);
	}
	});
	// ER
	adjustOffAndExperimentMetric(
	hour,
	new ExperimentHourMetric() {
	@Override
	short getSeconds() {
	return hour.getErSeconds();
	}
	@Override
	void setSeconds(short seconds) {
	hour.setErSeconds(seconds);
	}
	});
	// PCC
	adjustOffAndExperimentMetric(
	hour,
	new ExperimentHourMetric() {
	@Override
	short getSeconds() {
	return hour.getPccSeconds();
	}
	@Override
	void setSeconds(short seconds) {
	hour.setPccSeconds(seconds);
	}
	});
	roundAcceleratorSet(hour);
	roundExperimenterSet(hour);
	}

The Hall A EXPERIMENTER PVs, for reference:

• ABU: HLA:bta_sec_w_bm_w_daq_h
• BANU: HLA:bta_sec_w_bm_wo_daq_h
• BNA: HLA:bta_sec_wo_bm_h
• ACC: HLA:bta_sec_conf_chg_h
• ER: HLA:bta_sec_er_h
• UED: HLA:bta_sec_enr_h
• PCC: HLA:bta_sec_cc_h
• Timestamp: HLA:bta_uxtime_h

Workaround: As per usual, the measured values can be safely ignored in cases like this. But that does mean counting house staff needs to track what is happening and manually record what happened in the separate reported table.

Frequency of issue: On first glance I'd say rare. This probably only happens on hours that split an OFF boundary.

[slominskir]

Discussed this with Scott.

Sounds like BTM needs to read from PV HL{hall}:bta_sec_hall_off_h instead of BOOMHL{hall}OFF_h. The former will exclusive OR with ACC. That should address the case when both ACC and OFF are counted simultaneously (need to be mutually exclusive). For OFF and UED/PCC/ER counting simultaneously a tweak to LLAPP may help.

Aside: Technically, the hall is OFF when ACC, so when it counts both it's kinda correct. But we need it mutually exclusive, so we'll treat ACC as a subset of OFF and it needs to be subtracted from OFF.

See: https://tasklists.jlab.org/swlist/tasks/116285

Note: Crew Chief timesheets do not have ACC, for each hall, and instead just OFF per hall. But Experimenter timesheets have both ACC and OFF. Glad I'm not a Crew Chief having to make the numbers make sense at the end of the shift...

[slominskir]

Ops is asking for this. Re-prioritizing to High.