All instruments: multiple IOC stalls

[Tom-Willemsen]

Multiple beamlines have reported that various IOCs stalled at 07:30 or so this morning. This affects both "soft" iocs (e.g. RUNCTRL, GWBLOCK) and IOCs that communicate with hardware (both serial and ethernet).

Current solution: restart IBEX server

Beamlines where we've restarted the server:

• MAPS
• GEM
• LET
• INES

Beamlines where we have asked them to restart the server at a convenient time:

• LARMOR
• IRIS

[ThomasLohnert]

Also had calls this morning which I though resolved by just restarting individual IOCs. This was before we realised there was a wider issue, so they might still be in an unhappy state.

MARI: lots of blocks showed disconnected, restarted GWBLOCK
IRIS: Eurotherm 01 did not update calibration file readback, restarted Eurotherm IOC

[Tom-Willemsen]

Attempt at diagnosis so far:

• We think this may be related to some errors which are coming up in the windows event log at the same sort of time on each instrument
• "Most" of the affected ioc still works - e.g. channel access, console commands, etc.
• It is possible for this issue to affect only some of the PVs within a single IOC
  • This may be because the issue kills only one scan thread within an IOC
• The time of the failure is between 07:00 and 09:00 on most instruments. Not all instruments suffer this problem at exactly the same time.

[davidkeymer]

INES reported inconsistencies between jaw gaps shown on the OPI and the relevant motor positions in the table. A restart of the IBEX server remedied the situation.

[DominicOram]

We think this may be related to some errors which are coming up in the windows event log at the same sort of time on each instrument

What's the error? Might be useful to know if I see it again whilst on call

[FreddieAkeroyd]

Error in system log related to WMI performance service having multiple restarts. It seems not all values on an IOC stop updating, so current theory is that the deviocstats module (which monitors some performance counters) had an issue and hung, this then hung the db scan thread (1 second?) that is was using, but channel access and things reading at other polling rates were OK. I have added a nagios alert for IOC time (as per devIocStats) stopping updating.

[Tom-Willemsen]

+1 POLARIS, TOSCA, OSIRIS this morning at ~0400 hrs

[DominicOram]

A rather thrown together script that should pick up any IOCs that are out of date:

import mysql.connector
from genie_python.genie_cachannel_wrapper import CaChannelWrapper
from genie_python.channel_access_exceptions import UnableToConnectToPVException, ReadAccessException
import json
import zlib
from datetime import datetime

def get_value(pv):
    try:
        print("Getting {}".format(pv))
        return CaChannelWrapper.get_pv_value("{}".format(pv, to_string=True))
    except (UnableToConnectToPVException, ReadAccessException):
        print("Warning, couldn't get {}".format(pv))
        return None


def get_inst_list():
    pv_value = get_value("CS:INSTLIST")
    decompressed_json = zlib.decompress(pv_value.decode('hex'))
    return {} if pv_value is None else json.loads(decompressed_json)

inst_list = get_inst_list()

bad_iocs = []
bad_instruments = []

for instrument in inst_list:
    try:
        if instrument['isScheduled']:
            print(instrument)
            mydb = mysql.connector.connect(
              host=instrument['hostName'],
              user="report",
              passwd="$report",
              database="iocdb"
            )

            cursor = mydb.cursor()

            cursor.execute("SELECT iocname FROM iocdb.iocrt WHERE running = 1;")

            for ioc in cursor:
                try:
                    ioc_time = get_value("{}CS:IOC:{}:DEVIOS:TOD".format(instrument['pvPrefix'], ioc[0]))
                    datetime_object = datetime.strptime(ioc_time, '%m/%d/%Y %I:%M:%S')
                    diff = datetime.now() - datetime_object
                    print("{} {}: {}".format(instrument['pvPrefix'], ioc, datetime_object))
                    if diff.seconds > 30:
                        bad_iocs.append((instrument['pvPrefix'], ioc, datetime_object))
                        print("WARNING, {} {} is out of date".format(instrument['pvPrefix'], ioc, datetime_object))
                except:
                    bad_iocs.append((instrument['pvPrefix'], ioc, datetime_object))
    except:
        bad_instruments.append(instrument['pvPrefix'])


print("Out of date iocs: {}".format(bad_iocs))
print("Instruments not connected: ".format(bad_instruments))

[DominicOram]

It picked up potential issues on IMAT, OSIRIS, MARI

[FreddieAkeroyd]

On MARI it is still the issue from last week - they have been emailed about restarting