`smurf.stream` does not wait fully until streaming is running before returning

[BrianJKoopman]

After starting, the stream function checks that the ocs process has started:

sorunlib/src/sorunlib/smurf.py

Lines 365 to 371 in d5760b8

	for smurf in run.CLIENTS['smurf']:
	smurf.stream.start(subtype=subtype, tag=tag, kwargs=kwargs)
	for smurf in run.CLIENTS['smurf']:
	resp = smurf.stream.status()
	try:
	check_started(smurf, resp, timeout=120)

This check currently just checks for the ocs OpCode for 'running':

sorunlib/src/sorunlib/_internal.py

Lines 111 to 132 in d5760b8

	op_code = response.session.get('op_code')
	if op_code == 2: # STARTING
	_operation = client.__getattribute__(op)
	# Wait at most ~timeout seconds while checking the status
	for i in range(timeout):
	response = _operation.status()
	op_code = response.session.get('op_code')
	if op_code == 3:
	# Tricky to change state during testing w/little reward
	return # pragma: no cover
	time.sleep(1)
	error = f"Check timed out. Operation {op} in Agent {instance} stuck in " + \
	"'starting' state.\n" + str(response)
	raise RuntimeError(error)
	if op_code != 3: # RUNNING
	error = f"Operation {op} in Agent {instance} is not 'running'.\n" + \
	f"Current OpCode: {op_code}\n" + str(response)
	raise RuntimeError(error)

The process 'running' doesn't mean the data stream is actually on yet though, given simonsobs/socs#897. Once simonsobs/socs#897 is implemented, we will need to check that the data stream is actually on before returning from smurf.stream().

Until then, we're assuming data has started when beginning operations. This is likely affecting every operation that streams, then commands another piece of hardware. Examples:

• Stimulator
• Wiregrid
• HWP
• seq.scan

With the agent functionality added in simonsobs/socs#897, this should be entirely fixable in the internal logic, so I don't expect we'll need to modify the various hardware modules.