`curl_easy_setopt: 4` when using Pkg

[alex180500]

In windows with a fresh install of juliaup with Julia 1.10.3 when using any command in the Pkg REPL gives out this message:

┌ Error: curl_easy_setopt: 4
└ @ Downloads.Curl C:\Users\aless\.julia\juliaup\julia-1.10.3+0.x64.w64.mingw32\share\julia\stdlib\v1.10\Downloads\src\Curl\utils.jl:50

Everything is working correctly but there is this error continuously

[StefanKarpinski]

That error code indicates CURLE_NOT_BUILT_IN. Do you have http_proxy or https_proxy environment variables set? Or if not, do you have any git configs set to override protocols?

[alex180500]

By doing gci env:* | sort-object name in powershell I cannot see any http_proxy environment. My git config files are (git config --list):

diff.astextplain.textconv=astextplain
filter.lfs.clean=git-lfs clean -- %f
filter.lfs.smudge=git-lfs smudge -- %f
filter.lfs.process=git-lfs filter-process
filter.lfs.required=true
http.sslbackend=openssl
http.sslcainfo=C:/Program Files/Git/mingw64/etc/ssl/certs/ca-bundle.crt
core.autocrlf=true
core.fscache=true
core.symlinks=false
core.editor=notepad
pull.rebase=false
credential.helper=manager
credential.https://dev.azure.com.usehttppath=true
init.defaultbranch=master
user.email=REDACTED
user.signingkey=REDACTED
commit.gpgsign=true
tag.gpgsign=true
gpg.program=C:\Program Files (x86)\GnuPG\bin\gpg.exe

Also by doing

using Libdl
filter!(contains("curl"), dllist())

I get this:

"C:\\Users\\aless\\.julia\\juliaup\\julia-1.10.3+0.x64.w64.mingw32\\bin\\libcurl-4.dll"

[StefanKarpinski]

libcurl-4.dll seems likely to be the correct version of the library since it's the one we ship and this seems to be working for most people. Can you try this in the REPL:

using Downloads
Downloads.Curl.CURL_VERSION_STR
Downloads.download("https://pkg.julialang.org/registries", verbose=true)

[alex180500]

First of all sorry if I made a mistake on opening too many issues. Then thanks for the help.
This is the output from the REPL.

julia> using Downloads

julia> Downloads.Curl.CURL_VERSION_STR
"libcurl/8.4.0 Schannel zlib/1.2.13 libssh2/1.11.0 nghttp2/1.52.0"

julia> Downloads.download("https://pkg.julialang.org/registries", verbose=true)
┌ Error: curl_easy_setopt: 4
└ @ Downloads.Curl C:\Users\aless\.julia\juliaup\julia-1.10.3+0.x64.w64.mingw32\share\julia\stdlib\v1.10\Downloads\src\Curl\utils.jl:50
* WARNING: failed to open cookie file ""
* Couldn't find host pkg.julialang.org in the .netrc file; using defaults
* Hostname pkg.julialang.org was found in DNS cache
*   Trying [2a04:4e42::729]:443...
* Connected to pkg.julialang.org (2a04:4e42::729) port 443
* schannel: disabled automatic use of client certificate
* using HTTP/1.x
> GET /registries HTTP/1.1
Host: pkg.julialang.org
Accept: */*
User-Agent: curl/8.4.0 julia/1.10

* schannel: server closed the connection
< HTTP/1.1 301 EU internal redirect trigger
< Connection: close
< Content-Length: 0
< Server: Varnish
< Retry-After: 0
< Location: https://eu-central.pkg.julialang.org/registries
< x-geo-continent: EU
< x-geo-country: IT
< x-geo-region: GO
< Accept-Ranges: bytes
< Date: Wed, 29 May 2024 22:55:41 GMT
< Via: 1.1 varnish
< X-Served-By: cache-mxp6960-MXP
< X-Cache: HIT
< X-Cache-Hits: 0
< X-Timer: S1717023341.111815,VS0,VE0
<
* Closing connection
* schannel: shutting down SSL/TLS connection with pkg.julialang.org port 443
* Issue another request to this URL: 'https://eu-central.pkg.julialang.org/registries'
* Couldn't find host eu-central.pkg.julialang.org in the .netrc file; using defaults
* Found bundle for host: 0x185fba58320 [serially]
* Can not multiplex, even if we wanted to
* Re-using existing connection with host eu-central.pkg.julialang.org
> GET /registries HTTP/1.1
Host: eu-central.pkg.julialang.org
Accept: */*
User-Agent: curl/8.4.0 julia/1.10

< HTTP/1.1 302 Moved Temporarily
< Server: nginx/1.26.0
< Date: Wed, 29 May 2024 22:55:41 GMT
< Transfer-Encoding: chunked
< Connection: keep-alive
< Location: /registries.conservative
< X-lb-strategy: pkgservers_roundrobin
<
* Ignoring the response-body
* Connection #1 to host eu-central.pkg.julialang.org left intact
* Issue another request to this URL: 'https://eu-central.pkg.julialang.org/registries.conservative'
* Couldn't find host eu-central.pkg.julialang.org in the .netrc file; using defaults
* Found bundle for host: 0x185fba58320 [serially]
* Can not multiplex, even if we wanted to
* Re-using existing connection with host eu-central.pkg.julialang.org
> GET /registries.conservative HTTP/1.1
Host: eu-central.pkg.julialang.org
Accept: */*
User-Agent: curl/8.4.0 julia/1.10

< HTTP/1.1 200 OK
< Server: nginx/1.26.0
< Date: Wed, 29 May 2024 22:55:41 GMT
< Content-Type: application/octet-stream
< Content-Length: 88
< Connection: keep-alive
< Last-Modified: Wed, 29 May 2024 22:43:47 GMT
< ETag: "6657afa3-58"
< Accept-Ranges: bytes
< X-lb-strategy: pkgservers_roundrobin
<
* Connection #1 to host eu-central.pkg.julialang.org left intact
"C:\\Users\\aless\\AppData\\Local\\Temp\\jl_TcXmhkFutW"

[alex180500]

Any news? issue is still ongoing, I want to help but I cannot figure out what is the problem...

[StefanKarpinski]

So I have no idea what is happening here. You seem to have the correct version of libcurl that we ship. Everything seems to be working correctly as well. But you're getting those error messages indicating that a curl option is being set that libcurl doesn't understand. I'll have to think about what to do next to debug this...

[StefanKarpinski]

Ok, please try this:

using Downloads
@eval Downloads.Curl function setopt(easy::Easy, option::Integer, value)
     puts("setopt: $option = $value")
     @check curl_easy_setopt(easy.handle, option, value)
end
Downloads.download("https://pkg.julialang.org/registries")

This monkey-patches the definition of Downloads.Curl.setopt to print out the option and value when it gets called. I'm trying to narrow down which option is triggering the warning.

[alex180500]

This is the result of the script provided (thank you by the way!!!!)

setopt: 10103 = Ptr{Nothing} @0x0000024d0aef8320
setopt: 10010 = Ptr{UInt8} @0x0000024d0ab88818
setopt: 20312 = Ptr{Nothing} @0x0000024d4eaf39e0
setopt: 10313 = Ptr{Nothing} @0x0000024d0aef8320
setopt: 20079 = Ptr{Nothing} @0x0000024d4eaf3c80
setopt: 10029 = Ptr{Nothing} @0x0000024d0aef8320
setopt: 20011 = Ptr{Nothing} @0x0000024d4eaf3ee0
setopt: 10001 = Ptr{Nothing} @0x0000024d0aef8320
setopt: 20219 = Ptr{Nothing} @0x0000024d4eaf4140
setopt: 10057 = Ptr{Nothing} @0x0000024d0aef8320
setopt: 99 = true
setopt: 52 = true
setopt: 68 = 50
setopt: 161 = 7
setopt: 10018 = curl/8.4.0 julia/1.10
setopt: 51 = 1
setopt: 10031 =
setopt: 216 = 8
setopt: 78 = 30
setopt: 20 = 20
setopt: 19 = 1
setopt: 10153 = C:\Users\aless\.ssh\id_rsa
setopt: 10152 = C:\Users\aless\.ssh\id_rsa.pub
setopt: 10026 = Ptr{Nothing} @0x0000000000000000
setopt: 10002 = https://pkg.julialang.org/registries
setopt: 64 = true
setopt: 10183 = C:\Users\aless\.ssh\known_hosts
setopt: 13 = 0
setopt: 41 = false
setopt: 20094 = Ptr{Nothing} @0x0000000000000000
setopt: 10095 = Ptr{Nothing} @0x0000000000000000
setopt: 10023 = Ptr{Downloads.Curl.curl_slist_t} @0x0000024d7d7844a0
setopt: 44 = false
"C:\\Users\\aless\\AppData\\Local\\Temp\\jl_AhgFAPMHkN"

[giordano]

@StefanKarpinski side note, maybe setopt could print those things with @debug macros?

[StefanKarpinski]

Does it not show the error anymore when you do that? I was hoping for a setopt message right before the error and then we could infer that the option printed immediately prior was the one it's complaining about.

[alex180500]

I just noted one thing, if this code here:

using Downloads
@eval Downloads.Curl function setopt(easy::Easy, option::Integer, value)
     puts("setopt: $option = $value")
     @check curl_easy_setopt(easy.handle, option, value)
end
Downloads.download("https://pkg.julialang.org/registries")

is run from either the julia REPL or in the terminal by calling julia file.jl I get in the output this thing over here

setopt: 10103 = Ptr{Nothing} @0x0000021827b24240
setopt: 10010 = Ptr{UInt8} @0x0000021827c37c98
setopt: 20312 = Ptr{Nothing} @0x00000218294d5600
setopt: 10313 = Ptr{Nothing} @0x0000021827b24240
setopt: 20079 = Ptr{Nothing} @0x00000218294d58a0
setopt: 10029 = Ptr{Nothing} @0x0000021827b24240
setopt: 20011 = Ptr{Nothing} @0x00000218294d5b00
setopt: 10001 = Ptr{Nothing} @0x0000021827b24240
setopt: 20219 = Ptr{Nothing} @0x00000218294d5d60
setopt: 10057 = Ptr{Nothing} @0x0000021827b24240
setopt: 99 = true
setopt: 52 = true
setopt: 68 = 50
setopt: 161 = 7
setopt: 10018 = curl/8.4.0 julia/1.10
setopt: 51 = 1
setopt: 10031 =
setopt: 216 = 8
setopt: 78 = 30
setopt: 20 = 20
setopt: 19 = 1
setopt: 10153 = C:\Users\aless\.ssh\id_rsa
setopt: 10152 = C:\Users\aless\.ssh\id_rsa.pub
setopt: 10026 = Ptr{Nothing} @0x0000000000000000
setopt: 10002 = https://pkg.julialang.org/registries
setopt: 64 = true
setopt: 10183 = C:\Users\aless\.ssh\known_hosts
setopt: 13 = 0
setopt: 41 = false
setopt: 20094 = Ptr{Nothing} @0x0000000000000000
setopt: 10095 = Ptr{Nothing} @0x0000000000000000
setopt: 10023 = Ptr{Downloads.Curl.curl_slist_t} @0x00000218524b46d0
setopt: 44 = false
setopt: 10097 = C:\Users\aless\miniforge3\envs\quarto\Library\ssl\certs
┌ Error: curl_easy_setopt: 4
└ @ Downloads.Curl C:\Users\aless\.julia\juliaup\julia-1.10.4+0.x64.w64.mingw32\share\julia\stdlib\v1.10\Downloads\src\Curl\utils.jl:50

And the error is present.

But if I run from VS Code using the Julia: Execute active File in REPL I get no error. Actually, I just need to start the repl in VS Code and I can just do ]update and I also get no error.

Is there something related with Powershell? I'm using Powershell 7.4.2 in Windows Terminal 1.20.11381.0.
I will update now to Powershell 7.4.3 so I'll update soon.

[StefanKarpinski]

Do you have one of the following environment variables set?

• SSL_CERT_FILE
• SSL_CERT_DIR
• JULIA_SSL_CA_ROOTS_PATH

[visr]

Yeah this error can be reduced on Windows to setting SSL_CERT_DIR to an empty folder.

mkdir empty-certs
$Env:SSL_CERT_DIR = "empty-certs"
julia

The Conda activation script of OpenSSL started setting SSL_CERT_FILE and SSL_CERT_DIR on Windows since conda-forge/openssl-feedstock#157. So if you start Julia from an activated Conda environment with OpenSSL on Windows, you will run into this. And having OpenSSL in your Conda environment is likely, since also Python depends on it.

The easiest way to reproduce is by installing pixi, go to an empty folder and run:

D:\temp\act
❯ pixi init
✔ Initialized project in D:\temp\act\.

D:\temp\act
❯ pixi add openssl
✔ Added openssl

D:\temp\act
❯ pixi run julia
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.10.4 (2024-06-04)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> ENV["SSL_CERT_FILE"]
"D:\\temp\\act\\.pixi\\envs\\default\\Library\\ssl\\cacert.pem"

julia> ENV["SSL_CERT_DIR"]
"D:\\temp\\act\\.pixi\\envs\\default\\Library\\ssl\\certs"

julia> using Downloads

julia> @eval Downloads.Curl function setopt(easy::Easy, option::Integer, value)
            puts("setopt: $option = $value")
            @check curl_easy_setopt(easy.handle, option, value)
       end
setopt (generic function with 2 methods)

julia> Downloads.download("https://pkg.julialang.org/registries")
setopt: 10103 = Ptr{Nothing} @0x000001e33a183030
setopt: 10010 = Ptr{UInt8} @0x000001e33a11fde8
setopt: 20312 = Ptr{Nothing} @0x000001e33c8e3790
setopt: 10313 = Ptr{Nothing} @0x000001e33a183030
setopt: 20079 = Ptr{Nothing} @0x000001e33c8e3a10
setopt: 10029 = Ptr{Nothing} @0x000001e33a183030
setopt: 20011 = Ptr{Nothing} @0x000001e33c8e3c60
setopt: 10001 = Ptr{Nothing} @0x000001e33a183030
setopt: 20219 = Ptr{Nothing} @0x000001e33c8e3eb0
setopt: 10057 = Ptr{Nothing} @0x000001e33a183030
setopt: 99 = true
setopt: 52 = true
setopt: 68 = 50
setopt: 161 = 7
setopt: 10018 = curl/8.4.0 julia/1.10
setopt: 51 = 1
setopt: 10031 =
setopt: 216 = 8
setopt: 78 = 30
setopt: 20 = 20
setopt: 19 = 1
setopt: 10153 = C:\Users\visser_mn\.ssh\id_rsa
setopt: 10152 = C:\Users\visser_mn\.ssh\id_rsa.pub
setopt: 10026 = Ptr{Nothing} @0x0000000000000000
setopt: 10002 = https://pkg.julialang.org/registries
setopt: 64 = true
setopt: 10183 = C:\Users\visser_mn\.ssh\known_hosts
setopt: 13 = 0
setopt: 41 = false
setopt: 20094 = Ptr{Nothing} @0x0000000000000000
setopt: 10095 = Ptr{Nothing} @0x0000000000000000
setopt: 10023 = Ptr{Downloads.Curl.curl_slist_t} @0x000001e3454d8cc0
setopt: 44 = false
setopt: 10097 = D:\temp\act\.pixi\envs\default\Library\ssl\certs
┌ Error: curl_easy_setopt: 4
└ @ Downloads.Curl C:\Users\visser_mn\.julia\juliaup\julia-1.10.4+0.x64.w64.mingw32\share\julia\stdlib\v1.10\Downloads\src\Curl\utils.jl:50
"C:\\Users\\VISSER~1\\AppData\\Local\\Temp\\jl_9ov2H0nMbM"

The pixi run julia command runs this OpenSSL Conda activation script before starting Julia, which sets the environment variables. The openssl package relies on the ca-certificates package:

❯ pixi list
Package          Version       Build        Size       Kind   Source
ca-certificates  2024.6.2      h56e8100_0   152.9 KiB  conda  ca-certificates-2024.6.2-h56e8100_0.conda
openssl          3.3.1         h2466b09_1   8 MiB      conda  openssl-3.3.1-h2466b09_1.conda
ucrt             10.0.22621.0  h57928b3_0   1.2 MiB    conda  ucrt-10.0.22621.0-h57928b3_0.tar.bz2
vc               14.3          h8a93ad2_20  17 KiB     conda  vc-14.3-h8a93ad2_20.conda
vc14_runtime     14.40.33810   ha82c5b3_20  734.3 KiB  conda  vc14_runtime-14.40.33810-ha82c5b3_20.conda

And the SSL_CERT_DIR that it sets under Library\ssl\certs only contains an empty .keep file. By modifying the activation script locally I found that setting SSL_CERT_FILE does not seem to cause an issue, but setting an empty SSL_CERT_DIR does.

[StefanKarpinski]

What do you think the best mitigation would be? Should we check if the directory is empty and not apply the option if it is?