State Handler Doesn’t Respect Retry Signals

[marvin-robot]

Opened from the Prefect Public Slack Community

jeffbrainerd: Hello Prefect team, a question about retrying tasks conditionally. In our task state handler we look at the failure message every time a task fails and we want to retry on some of those errors. Per this doc entry:
https://docs.prefect.io/core/concepts/execution.html#state-signals we perform a raise signals.RETRY in those cases. Problem is this does not appear to work in the task handler. This code gets called:

prefect/src/prefect/engine/cloud/task_runner.py

Lines 96 to 105 in 5b7015a

	try:
	new_state = super().call_runner_target_handlers(
	old_state=old_state, new_state=new_state
	)
	except Exception as exc:
	msg = "Exception raised while calling state handlers: {}".format(repr(exc))
	self.logger.exception(msg)
	if raise_on_exception:
	raise exc
	new_state = Failed(msg, result=exc)

and the task ultimately is failed, not retried. So should our code be something more like this:
return Retrying(msg=msg, context=new_state.context)
or should this in fact work? Thanks!

dylan: Hey <@USBCBEJDV>!

Great question, let me take a look

dylan: Hey <@USBCBEJDV>

dylan: This is a bug! Thank you for finding it

dylan: I’ll open an issue 👍

dylan: <@ULVA73B9P> issue “State Handler Doesn’t Respect Retry Signals”

dylan: <@ULVA73B9P> open “State Handler Doesn’t Respect Retry Signals”

Original thread can be found here.

[jeffbrainerd]

Thanks for taking care of this! 🙏