BUG: Race Condition in socket_callback #272
Description
When curl_multi_socket_action is called in the task created by socket_callback, sometimes the callback has not finished and curl has not internally called set_in_callback(multi, FALSE) yet. This causes the curl_multi_socket_action to fail and often causes connections to hang or fail. This affects concurrent use of gRPCClient.jl, basically any concurrent use of it with async quickly results in this issue: JuliaComputing/gRPCClient.jl#42
The error message associated with this is:
┌ Error: curl_multi_socket_action: 8
└ @ Downloads.Curl ~/.virtualenvs/full-solution/julia_env/pyjuliapkg/install/share/julia/stdlib/v1.11/Downloads/src/Curl/utils.jl:57
Code 8 is CURLM_RECURSIVE_API_CALL, which is telling us that we are using the multi handle before a callback for it has completed.
Interestingly enough it appears that at the time of the failure if we constantly retry curl_multi_socket_action without yielding to the event loop it will keep failing forever. However, if we yield back to the event loop between retries, it appears that control is actually passed back to curl at some point and the callback flag is cleared.
Here is a snippet fixing the race condition in a bit of a hackish way. I'm putting together a pull request but I'm not sure if this is the right place to do it since it appears Downloads.jl is now a part of the standard library.
lock(multi.lock) do
watcher.readable || watcher.writable || return # !isopen
while true
status = curl_multi_socket_action(multi.handle, sock, flags)
if status != 8
break
end
# Yield back to the event loop to let Julia return control back to curl
sleep(0.0)
end
check_multi_info(multi)
end