"[48] An unknown option was passed in to libcurl" on CentOS 7

[EliahKagan]

Current behavior 😯

On CentOS 7, if gitoxide is built with libcurl-devel installed (which provides header files for libcurl), then the resulting binaries break when attempting to use curl. The two most readily observable effects are:

• Cloning https:// URLs with gix clone fails.
• gix-transport::blocking-transport-http-only tests fail.

This only happens if libcurl-devel is installed. As detailed below, it seems that this causes gitoxide to be built against the old version of libcurl, even though that version does not have the features gitoxide requires. If libcurl-devel is not installed, then gitoxide is built without attempting to use that old version, and it does work. Either way, other functionality, including clones from SSH URLs, still works.

CentOS 7 is one of the oldest supported releases of any GNU/Linux distribution. That, taken together with the notability of CentOS and the absence of any need to subscribe to get backported fixes for newly discovered security vulnerabilities, makes CentOS 7 a useful operating system for testing version compatibility.

Although CentOS 7 has also been important for its own sake, that is less the case now since, as of this writing, it only has about one month of support remaining. However, I would be concerned that this effect might also occur on other systems in significant use (some of which would be harder for me to test).

Whether the situation where the system has an old version of libcurl as well as headers for that version installed is worth supporting or not remains a reasonable question. But I did not have to take any special effort to cause this to occur--I happened to have those headers installed because I needed them to build Git--and no error or warning was ever issued about version incompatibility.

This happens both with gitoxide 0.36.0 as installed with cargo install gitoxide and with the version installed from the tip of the main branch by running cargo install --path . in the cloned gitoxide repository. Both give this output:

ek in 🌐 Eald in ~/src
❯ gix clone https://github.com/Byron/gitoxide.git
Error: [48] An unknown option was passed in to libcurl

Changing the URL to the SSH URL git@github.com:Byron/gitoxide.git works.

As for the automated tests, full test run output can be seen here, which is part of this gist, which also has some other logs, such as this shorter version. The failing tests are:

     Summary [  72.668s] 2364 tests run: 2354 passed (1 leaky), 10 failed, 2 skipped
        FAIL [   0.237s] gix-transport::blocking-transport-http-only client::blocking_io::http::check_content_type_is_case_insensitive
        FAIL [   0.016s] gix-transport::blocking-transport-http-only client::blocking_io::http::clone_v1
        FAIL [   0.022s] gix-transport::blocking-transport-http-only client::blocking_io::http::handshake_and_lsrefs_and_fetch_v2
        FAIL [   0.018s] gix-transport::blocking-transport-http-only client::blocking_io::http::handshake_and_lsrefs_and_fetch_v2_googlesource
        FAIL [   0.019s] gix-transport::blocking-transport-http-only client::blocking_io::http::handshake_and_lsrefs_and_fetch_v2_service_announced
        FAIL [   0.016s] gix-transport::blocking-transport-http-only client::blocking_io::http::handshake_v1
        FAIL [   0.148s] gix-transport::blocking-transport-http-only client::blocking_io::http::http_authentication_error_can_be_differentiated_and_identity_is_transmitted
        FAIL [   0.133s] gix-transport::blocking-transport-http-only client::blocking_io::http::http_error_results_in_observable_error
        FAIL [   0.128s] gix-transport::blocking-transport-http-only client::blocking_io::http::http_status_500_is_communicated_via_special_io_error
        FAIL [   0.135s] gix-transport::blocking-transport-http-only client::blocking_io::http::http_will_use_pipelining

The failures look like:

test client::blocking_io::http::clone_v1 ... FAILED

failures:

failures:
    client::blocking_io::http::clone_v1

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 17 filtered out; finished in 0.00s


--- STDERR:              gix-transport::blocking-transport-http-only client::blocking_io::http::clone_v1 ---
Error: Http(Detail { description: "[48] An unknown option was passed in to libcurl" })

Or, for some:

test client::blocking_io::http::http_authentication_error_can_be_differentiated_and_identity_is_transmitted ... FAILED

failures:

failures:
    client::blocking_io::http::http_authentication_error_can_be_differentiated_and_identity_is_transmitted

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 17 filtered out; finished in 0.12s


--- STDERR:              gix-transport::blocking-transport-http-only client::blocking_io::http::http_authentication_error_can_be_differentiated_and_identity_is_transmitted ---
thread 'client::blocking_io::http::http_authentication_error_can_be_differentiated_and_identity_is_transmitted' panicked at gix-transport/tests/client/blocking_io/http/mod.rs:34:28:
no source() in: Http(Detail { description: "[48] An unknown option was passed in to libcurl" })
stack backtrace:
   0:     0x55ebab12cb62 - std::backtrace_rs::backtrace::libunwind::trace::he4ee80166a02c846
                               at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/../../backtrace/src/backtrace/libunwind.rs:105:5
   1:     0x55ebab12cb62 - std::backtrace_rs::backtrace::trace_unsynchronized::h476faccf57e88641
                               at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2:     0x55ebab12cb62 - std::sys_common::backtrace::_print_fmt::h430c922a77e7a59c
                               at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/sys_common/backtrace.rs:68:5
   3:     0x55ebab12cb62 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::hffecb437d922f988
                               at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/sys_common/backtrace.rs:44:22
   4:     0x55ebab15795c - core::fmt::rt::Argument::fmt::hf3df69369399bfa9
                               at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/fmt/rt.rs:142:9
   5:     0x55ebab15795c - core::fmt::write::hd9a8d7d029f9ea1a
                               at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/fmt/mod.rs:1153:17
   6:     0x55ebab129a0f - std::io::Write::write_fmt::h0e1226b2b8d973fe
                               at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/io/mod.rs:1843:15
   7:     0x55ebab12c934 - std::sys_common::backtrace::_print::hd2df4a083f6e69b8
                               at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/sys_common/backtrace.rs:47:5
   8:     0x55ebab12c934 - std::sys_common::backtrace::print::he907f6ad7eee41cb
                               at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/sys_common/backtrace.rs:34:9
   9:     0x55ebab12e5eb - std::panicking::default_hook::{{closure}}::h3926193b61c9ca9b
  10:     0x55ebab12e343 - std::panicking::default_hook::h25ba2457dea68e65
                               at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:292:9
  11:     0x55ebab12ea8d - std::panicking::rust_panic_with_hook::h0ad14d90dcf5224f
                               at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:779:13
  12:     0x55ebab12e962 - std::panicking::begin_panic_handler::{{closure}}::h4a1838a06f542647
                               at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:657:13
  13:     0x55ebab12d036 - std::sys_common::backtrace::__rust_end_short_backtrace::h77cc4dc3567ca904
                               at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/sys_common/backtrace.rs:171:18
  14:     0x55ebab12e694 - rust_begin_unwind
                               at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:645:5
  15:     0x55ebaae868c5 - core::panicking::panic_fmt::h940d4fd01a4b4fd1
                               at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/panicking.rs:72:14
  16:     0x55ebaaecd488 - blocking_transport_http_only::client::blocking_io::http::assert_error_status::{{closure}}::h4cdff2fa0236af1b
                               at /home/ek/repos/gitoxide/gix-transport/tests/client/blocking_io/http/mod.rs:34:28
  17:     0x55ebaaee6d6c - core::option::Option<T>::unwrap_or_else::h4c09bdd3572b7f53
                               at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/option.rs:978:21
  18:     0x55ebaaed4350 - blocking_transport_http_only::client::blocking_io::http::assert_error_status::h0f8cbf29fe36aaa5
                               at /home/ek/repos/gitoxide/gix-transport/tests/client/blocking_io/http/mod.rs:32:17
  19:     0x55ebaaed84c8 - blocking_transport_http_only::client::blocking_io::http::http_authentication_error_can_be_differentiated_and_identity_is_transmitted::h4cc6ef8bcfd4d1ac
                               at /home/ek/repos/gitoxide/gix-transport/tests/client/blocking_io/http/mod.rs:150:32
  20:     0x55ebaaecdfe7 - blocking_transport_http_only::client::blocking_io::http::http_authentication_error_can_be_differentiated_and_identity_is_transmitted::{{closure}}::h892c8be9af307e11
                               at /home/ek/repos/gitoxide/gix-transport/tests/client/blocking_io/http/mod.rs:149:85
  21:     0x55ebaae96206 - core::ops::function::FnOnce::call_once::h95bcc5f3b554b19a
                               at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/ops/function.rs:250:5
  22:     0x55ebaaf64fff - core::ops::function::FnOnce::call_once::hdf5ca86569853e72
                               at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/ops/function.rs:250:5
  23:     0x55ebaaf64fff - test::__rust_begin_short_backtrace::hd4329dc6408133c8
                               at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/test/src/lib.rs:621:18
  24:     0x55ebaaf64683 - test::run_test_in_process::{{closure}}::h28125576fa02ab1b
                               at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/test/src/lib.rs:644:60
  25:     0x55ebaaf64683 - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h263d7f9b738b9dba
                               at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/panic/unwind_safe.rs:272:9
  26:     0x55ebaaf64683 - std::panicking::try::do_call::h07c639bca005fe67
                               at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:552:40
  27:     0x55ebaaf64683 - std::panicking::try::h58a07e442d3bdb92
                               at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:516:19
  28:     0x55ebaaf64683 - std::panic::catch_unwind::h755d3ddccf2ee291
                               at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panic.rs:146:14
  29:     0x55ebaaf64683 - test::run_test_in_process::h1afe055b8c5de7fa
                               at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/test/src/lib.rs:644:27
  30:     0x55ebaaf64683 - test::run_test::{{closure}}::h3810503c3fff9a00
                               at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/test/src/lib.rs:567:43
  31:     0x55ebaaf2d283 - test::run_test::{{closure}}::he05fab8f6f4e5a70
                               at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/test/src/lib.rs:595:41
  32:     0x55ebaaf2d283 - std::sys_common::backtrace::__rust_begin_short_backtrace::he823da8fbe6df054
                               at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/sys_common/backtrace.rs:155:18
  33:     0x55ebaaf31def - std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}::h4ec5c7fc245f87d0
                               at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/thread/mod.rs:528:17
  34:     0x55ebaaf31def - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::hc69a28560447662e
                               at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/panic/unwind_safe.rs:272:9
  35:     0x55ebaaf31def - std::panicking::try::do_call::hf528d3cff240a786
                               at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:552:40
  36:     0x55ebaaf31def - std::panicking::try::hb11bf079f9b1f405
                               at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:516:19
  37:     0x55ebaaf31def - std::panic::catch_unwind::hcee8d50a8dc1d323
                               at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panic.rs:146:14
  38:     0x55ebaaf31def - std::thread::Builder::spawn_unchecked_::{{closure}}::hb48d60f284150728
                               at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/thread/mod.rs:527:30
  39:     0x55ebaaf31def - core::ops::function::FnOnce::call_once{{vtable.shim}}::h04ac49919f9d987a
                               at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/ops/function.rs:250:5
  40:     0x55ebab132f75 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h19b9e642d37e7272
                               at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/alloc/src/boxed.rs:2020:9
  41:     0x55ebab132f75 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h97265befc434d3ae
                               at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/alloc/src/boxed.rs:2020:9
  42:     0x55ebab132f75 - std::sys::pal::unix::thread::Thread::new::thread_start::h420dad5cf01a9f35
                               at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/sys/pal/unix/thread.rs:108:17
  43:     0x7f53ac84cea5 - start_thread
  44:     0x7f53ac06fb0d - clone
  45:                0x0 - <unknown>

Sometimes this kind of error happens due to a mismatch in headers and the installed library, but that's not the case here. The versions of the relevant packages on the system are:

libcurl.x86_64                                           7.29.0-59.el7_9.2                                      @updates
libcurl-devel.x86_64                                     7.29.0-59.el7_9.2                                      @updates

The curl-related Rust crates built to run the tests were:

   Compiling curl-sys v0.4.72+curl-8.6.0

   Compiling curl v0.4.46

In case it's somehow relevant, the installed version of curl itself matches that of the installed libcurl and the installed libcurl headers, 7.29.0-59.el7_9.2.

As shown in the full log gist, the above tests were run by cleaning and issuing RUST_BACKTRACE=full cargo nextest run --all --no-fail-fast, and installed versions of system packages were checked using commands like yum list installed 'libcurl*'.

This might not be a bug in gitoxide. It may relate to some considerations discussed in alexcrichton/curl-rust#414. This kind of error message is also mentioned in alexcrichton/curl-rust#160 though that does not seem to be very related.

Expected behavior 🤔

When installing gitoxide, especially when building it from source as cargo install gitoxide does, the resulting binaries should support HTTP and HTTPS URLs. If that is not possible, then some indication should be given that they are not supported, preferably with information about how to install it in a way where they are supported.

Likewise, the tests should ideally pass.

Git behavior

Replacing gix with git in the clone command makes cloning from HTTPS URLs work, including when building the latest version of git from source on the system:

ek in 🌐 Eald in ~/src
❯ git --version
git version 2.45.1

ek in 🌐 Eald in ~/src
❯ git clone https://github.com/Byron/gitoxide.git
Cloning into 'gitoxide'...
remote: Enumerating objects: 145172, done.
remote: Counting objects: 100% (2946/2946), done.
remote: Compressing objects: 100% (1069/1069), done.
remote: Total 145172 (delta 1875), reused 2788 (delta 1820), pack-reused 142226
Receiving objects: 100% (145172/145172), 54.80 MiB | 1.53 MiB/s, done.
Resolving deltas: 100% (94940/94940), done.

ek in 🌐 Eald in ~/src
❯ /usr/bin/git --version
git version 1.8.3.1

ek in 🌐 Eald in ~/src
❯ /usr/bin/git clone https://github.com/Byron/gitoxide.git
Cloning into 'gitoxide'...
remote: Enumerating objects: 145172, done.
remote: Counting objects: 100% (2946/2946), done.
remote: Compressing objects: 100% (1070/1070), done.
remote: Total 145172 (delta 1875), reused 2787 (delta 1819), pack-reused 142226
Receiving objects: 100% (145172/145172), 54.88 MiB | 1.56 MiB/s, done.
Resolving deltas: 100% (94949/94949), done.

I built git by cloning the repository, checking out the tag v2.45.1, and following the build instructions with an all-default configuration except for the destination directory (and the amount of parallelism).

I ran the complete git test suite after building git, and there were no failures.

Steps to reproduce 🕹

Since CentOS 7 is old (but still, as of this writing, supported) and there are some special considerations in setting up tooling on it, this gives more detailed instructions than usual.

On CentOS 7 with Rust installed via rustup using default settings and a fresh clone of gitoxide or after cleaning as if by git restore . and gix clean -xde:

1. Install a recent version of cmake, which gitoxide requires to build. I installed cmake version 3.29.3 from source. (Its own test suite fully passed.) Other software, besides cmake and software obtained using rustup or cargo, can be from official CentOS 7 repositories.
2. Run cargo install cargo-nextest. (Using cargo quickinstall or cargo binstall do not work on CentOS 7, as noted below.)
3. Run cargo nextest run --all --no-fail-fast. Optionally set the RUST_BACKTRACE environment variable to 1 or full (I used full), e.g., RUST_BACKTRACE=full cargo nextest run --all --no-fail-fast.

That produces the unit test failures. To test gix clone with HTTPS:

1. Install gitoxide in the way most CentOS 7 users who install it to obtain the gix and ein commands would do so, by running cargo install gitoxide. The reason to do it this way is partly to ensure that the problem is not caused by an old dependency. But it is much more so because using cargo quickinstall or cargo binstall unfortunately do not work properly on CentOS due to a separate issue related to what versions of libc the prebuilt binaries support. See #1392 on that issue and why I believe it is separate.
2. Run gix clone https://github.com/Byron/gitoxide.git, observing the failure.
3. Uninstall gitoxide by running cargo uninstall gitoxide.
4. cd to the gitoxide repository directory, and install gitoxide from there using cargo install --path .
5. Run gix clone https://github.com/Byron/gitoxide.git again, observing the failure.
6. Optionally uninstall with cargo uninstall ..
7. Optionally repeat the clone with either the system-provided or a current build of git (using git instead of gix), observing that this succeeds.

Both in steps 2 and 5, the error message is exactly:

Error: [48] An unknown option was passed in to libcurl

[EliahKagan]

I just made a major revision to this issue description: the problem only occurs if the libcurl header files are installed. If they are not installed, everything works fine. See the revised issue description above for further details.

[Byron]

That's a great find, thanks a million for the perfect description of the issue and the research that went into it.

Even though as long as there is no wide-spread (and still maintained) system that also shows this issue I wouldn't want to go all in to this, there can be nothing wrong with gently probing if there are simple solutions.

On our side, all we can do is tune the features of the curl crate - maybe there is something that makes this work?

It's notable that even if one working curl feature(set) is found, it might still not be desirable to generally set them due to reduced compatibility elsewhere. However, it would be possible to pass-through support for turning these curl features so users can turn them on, on demand.

[EliahKagan]

Since the problem seems to be attempting to use the system-provided libcurl when its headers can be found during the build, I wonder if there is some way to prevent it from being used--and a fresh build of a new enough version used instead, if I am understanding correctly--unless it is in a version range where it is known to have the necessary features/options. I am not sure if that could happen here or if it would have to be done in the curl crate or elsewhere. It seems to me that this might be better than attempting to narrow the features that are used.

I suspect the problem described here could happen with slightly newer versions in other releases of GNU/Linux distributions that might remain supported even after next month, though as in #1392 it might be a bit harder for me to test that in setups that sufficiently resemble actual usage since subscriptions (e.g., Ubuntu Pro) are sometimes required to receive backported security fixes on such old versions.

I also wonder if this could happen in the future with a newer version of libcurl. If not, then whatever mechanism is used to avoid using a future version with incompatible changes could maybe be extended to avoid using an old version that does not yet support the necessary features.