LARMOR: error message from genie python

[FreddieAkeroyd]

At 13:23 a waitfor was started, at 17:20 and then every few minutes after that is now printed:

2020-11-11T17:20:48	(CMD)	(17808)	2020-11-11 17:20:48.374781: Exception in waitfor loop: UnableToConnectToPVException: Unable to connect to PV IN:LARMOR:DAE:GOODUAH: does not exist
2020-11-11T17:20:48	(CMD)	(17808)	2020-11-11 17:20:48.512496: Exception cleared

as of now (21:30) it is still printing

Acceptance Criteria

1. The root cause of the message is identified
2. The cause of the message is eliminated

[FreddieAkeroyd]

Occasionally the genie python log gives:

2020-11-11T17:34:56	(CA)	(17808)	ERROR: CAException: type=-1 state=200 op=5 file=c:\instrument\apps\epics\base\master\modules\ca\src\client\cac.cpp lineNo=1320

This is pvMultiplyDefinedNotify

[FreddieAkeroyd]

The gap between error messages started out at 30 seconds and has gradually been getting longer, it is now 10 minutes

[Tom-Willemsen]

+1 EMU
+1 POLARIS (probably)

[DominicOram]

Do we have any idea of a way forward on this? It's next on the ready column but I'm not sure how to pursue it.

[FreddieAkeroyd]

I sent email to instrument scientists suggesting they restart python console, this seems to have fixed it. Not sure what underlying issue is.

[Tom-Willemsen]

I've written a script on branch Ticket5893 of the private scripts repo which checks for occurences of this error on all instruments. It looks like it started happening on the 11th for all instruments and continued until they restarted genie_python. However the last few days have been error-free on all instruments.

Given that this feels like a hard-to-reproduce one off error, I think we should leave it at that at the moment and investigate further if it recurs.

Maybe we could add some variant of the above-mentioned script to nagios?

Also added to wiki at https://github.com/ISISComputingGroup/ibex_developers_manual/wiki/genie_python-Troubleshooting#repeated-error-messsages-in-console-while-waiting