After a certain amount of time, the on-behalf-of flow ceases to function with the logged in user's token. The error returned is invalid_grant error code 500133. As far as I can tell, this corresponds to the assertion being expired, but I don't know why EasyAuth isn't catching this on the outside (and denying access correspondingly). Logging out and back in fixes the issue.
It might be worth looking for this and sending the user to the login flow if the error code comes back this way.