Allow to opt-out of automatic msys2_mingw_dependency installation

[tknerr]

Hi @jcupitt ,

as discussed in #331 (see comments), we have the need of more fine-grained control on the libvips library version to be used.

When you install the ruby-vips gem on Windows, it would automatically (and unconditionally) install the latest available libvips dependency to your msys2 ruby devkit environment (see here in ruby-vips.gemspec).

There is the %RUBY_DLL_PATH% environment variable which you can point to an external libvips installation (e.g. obtained from libvips/build-win64-mxe releases), but this only works when there is no libvips installation present in the msys2 ruby devkit environment.

So in order to make use of %RUBY_DLL_PATH% to get control of the libvips version being installed, you need to make sure that there is no parallel libvips installation in the msys2 ruby devkit environment. But here's the catch: every time you re-install or update the ruby-vips gem, you will get an msys2 libvips installation as a side effect, breaking the use of the external libvips version in %RUBY_DLL_PATH%.

One solution to this could be to provide a possibility to opt-out of the installation of "msys2_mingw_dependencies" in the gemspec (e.g. via an env var like SKIP_MSYS2_MINGW_DEPENDENCIES=1 or so).

What do you think? Does that sound feasible / sensible? Do you have other suggestions on how to enable better control over the libvips library version?

Cheers,
Torben

[jcupitt]

Hi again,

Yes, this is a tricky one.

Another possibility would be to add some code to ruby-vips here:

https://github.com/libvips/ruby-vips/blob/master/lib/vips.rb#L13-L34

It could check for an env var called VIPSHOME and prepend that to the library name, perhaps:

def library_name(name, abi_number)
  lib = ''

  if ENV.has_key? 'VIPSHOME'
    lib += "#{ENV['VIPSHOME']}/lib/"
  end

  if FFI::Platform.windows?
    lib += "lib#{name}-#{abi_number}.dll"
  elsif FFI::Platform.mac?
    lib += "#{name}.#{abi_number}"
  else
    lib += "#{name}.so.#{abi_number}"
  end

  lib
end

Though some platforms use lib64 or lib/x86_64-linux-gnu/ which is annoying.

[tknerr]

Hi @jcupitt, thanks for the quick reply. That sounds even better! 👍🏻

1. because having a dedicated VIPSHOME env var makes it a first-class feature of "ruby-vips" (vs RUBY_DLL_PATH which is a feature of RubyInstaller which is quirky and not under our control, and also works for Windows only)
2. because that would then work cross-platform (i.e. can be used on Linux, Windows, MacOS in the same way), doesn't it?

If I get (or rather guess) it right on how ffi works, by explicitly loading the libraries from VIPSHOME via ffi_lib if present, there is also no chance this might interfere with an additionally installed libvips in msys2 / mingw, right? (and opting out of that would not even be necessary, then)

[jcupitt]

I'm honestly not sure :( We'd need to do some testing. Perhaps opening libvips-42.dll in c:\libvips-8.12.4\bin would load dependencies like libopenslide.dll from the mingw area by mistake? That could cause horrible crashes.

Thinking about it some more, really the best thing would be to fix the pacman package. Maybe open a bug report here?

https://github.com/msys2/MINGW-packages/issues?q=is%3Aissue+is%3Aopen+libvips

[kleisauke]

It seems that RubyInstaller honors the --ignore-dependencies option of gem:
https://github.com/oneclick/rubyinstaller2/blob/ee44ad03f22cedfccb368f33dc9081daa84d0d81/resources/files/operating_system.rb#L8

So, to avoid installing libvips from MSYS2, you can just do:

$ gem install ruby-vips ffi --ignore-dependencies

FWIW, I tried to debug that crash with GDB using the mingw-w64-ucrt-x86_64-gdb package, but without debug symbols it's difficult to debug that further.

[tknerr]

Hi @jcupitt , thanks for the link to the MINGW-packages repo, I didn't know where to report this 👍🏻 but yes, fixing the pacman package would help for #331 (the crash), but not for #332 (how to control libvips version) though...

For sure, I can support with testing the VIPSHOME approach sketched above, also checking if it accidentally picks up libraries from the wrong location or not. Is the snippet from above all there is to it, or are there more places that would need to be adapted? (e.g. does ruby-vips exclusively use the libraries, or does it expect the correct vips.exe on the PATH or so?). If you want, I can also create a PR for that to start with.

@kleisauke yes, thanks for the hint regarding --ignore-dependencies. I found that too while trying to trick around it, but usually we freeze + install our gem dependencies via bundler, and there this approach won't work :-/

[jcupitt]

Yes, that's the only thing you'd need to change (I think!). But messing with paths and library resolution is extremely fragile and we're likely to trigger a lot of difficult to debug crashes.

I was thinking about this again -- I now think your original idea was right and we should have an env var like RUBYVIPS_EXTERNAL_LIBVIPS to turn off msys2_mingw_dependencies.

Perhaps (untested):

    "msys2_mingw_dependencies" => ENV.key?("RUBYVIPS_EXTERNAL_LIBVIPS") ? "" : "libvips"

So to use an external libvips you'd need to set RUBYVIPS_EXTERNAL_LIBVIPS to stop installation of the msys one, and set RUBY_DLL_PATH to include your external libvips bin area.

Any thoughts?

[jcupitt]

I came across this interesting article from a few years ago:

https://bibwild.wordpress.com/2015/09/09/optional-gem-dependencies/

Which (I think) advocates removing optional dependencies and checking at runtime instead, perhaps with a helpful message.

@jrochkind sorry to ping you, but I wonder if you have any insights on this issue?

[jrochkind]

You can ping me anytime @jcupitt, I love vips and appreciate and am impressed by your work! (Although I am a rubyist and use vips, I don't currently use ruby-vips -- my code calls out to a shell calling command line vips currently!)

There is no way to "opt out" of something in a ruby gemspec at runtime -- any conditional logic in the gemspec will be applied at gem build/release time, once when jcupitts builds and releases the gem, not at individual installation runtime. So definitely don't try that, which I think is unfortuantely what you were settling on -- either something is in a gemspec or is not, there's no way to make it conditional, with the conditions applied to the individual installation. Having something in the gemspec dependent on an ENV variable wont' work -- it'll be calculated at gem build/release time, and then fixed in place.

I don't develop or run ruby on Windows though, so don't totally understand the windows-specific issues. Looking at the current ruby-vips gemspec... I see in gem metadata, the key/value "msys2_mingw_dependencies" => "libvips" -- I have never seen this before, from the discussion here it sounds like you can express windows OS-level dependencies in gem metadata, that are actually enforced at gem install time? I am not familiar with this feature at all! So it's hard for me to judge the pros/cons of different ways of doing things.

With ordinary ruby gemspec dependencies (in gemspec as add_runtime_dependency) -- which this is not -- there are definitely pro's and con's of making an optional dependency only checked at runtime. Mainly the downside is that you are no making available ordinary ruby dependency features to do things like automatically install all dependencies at installation time, or compute the latest version(s) of all dependencies that are mutually satisfactory to each other. But if including the dependency in the ordinary way causes problems for some users -- whose usage isn't going to require the dependency at all, but where having it causes them problems (generally either long installation time/sizes, or conflicts with their desired dependency tree) -- it's kind of the only option to avoid the universal dependency.

If you do have an "optional dependency" which is only enforced at runtime -- I think it's important for the runtime checking to be very robust and the error message very clear, including (when it comes to rubygems) expressing and reporting the range of versions of the optional dependency that would be acceptable. (Rails' own runtime optional checking is not as good as it could be here when it comes to reporting).

That's my thinking on ordinary ruby gemspec dependencies. But this msys2_mingw_dependency is not that -- it's a windows feature I didn't even realize existed! (In non-Windows OSs, ruby gems generally have no formal way to express non-ruby-gem OS-level dependencies at all, as far as I know!). So I'm not totally sure how it applies here.

[jcupitt]

Hi! Thanks for the feedback.

It sounds like the best thing is to remove msys2_mingw_dependencies (as you guessed, it's way for windows ruby users to install gems with binary components, which otherwise can be extremely difficult) and display the best error message we can instead.

Presumably a try / catch around the first ffi import, maybe here:

https://github.com/libvips/ruby-vips/blob/master/lib/vips.rb#L45

Does anyone fancy taking a stab at a prototype PR?

[jcupitt]

Although I am a rubyist and use vips, I don't currently use ruby-vips -- my code calls out to a shell calling command line vips currently!

Oh, interesting. Do you use rails? Does ActiveRecord not fit your use case well?

[jrochkind]

Oh, interesting. Do you use rails? Does ActiveRecord not fit your use case well?

I do use rails. I think maybe you're asking about "ActiveStorage"? It's true I don't use ActiveStorage (I do use ActiveRecord, but I don't think it's relevant here), I do my own file handling and transformation with shrine. I could still use the image_processing gem that ActiveStorage uses (which is actually by janko, the same author as shrine) -- but I don't. Partially because some of this code pre-dates ActiveStorage/ImageProcessing, partially because file handling is central to my app and I just want complete control, including of things that may or may not be available via ImageProcessing.

I definitely could still use ruby-vips but... I just don't? I just felt more comfortable not relying on anything but the command line, and not having to worry about any possible concurrency or ruby memory bugs. it's possible i'll switch to ruby-vips in the future but it's working fine.

[jrochkind]

which otherwise can be extremely difficult

I now worry your proposal would make things "extremely difficult" for the common case, for the benefit of making something more convenient that's an edge case.

But I don't use ruby on Windows, I don't fully understand the case here (as far as I know, on non-windows there isn't even a facility to formally express non-ruby dependencies like this, it's unique to windows!), so I'll stay out of any further discussion!

[jcupitt]

Ah! I had an idea!

How about removing the mingw dependency from ruby-vips, and making a tiny new gem called ruby-vips-windows (or maybe ruby-vips-mingw? something like that?) which depends on ruby-vips, and which has the mingw dependency.

Now windows users who want the mingw binary can put ruby-vips-mingw in their gemspec, and everyone else can use plain ruby-vips.

I suppose the downside is that the ruby-vips-mingw version numbers will need to exactly match the ruby-vips ones so people can still pin etc. Can we have two gems in one repo and generate the second programmatically?

[jrochkind]

@jcupitt

That works to do what you say, and works for people who are including ruby-vips in an app.

The problem is -- what if you are writing a gem, and want to declare ruby-vips as a dependency of that gem? You are writing a gem that works on any platform, so you aren't going to declare ruby-vips-windows as a dependency. Or wait, would you? I mean, it would work fine mostly just as it is now -- except for the problem at the top of this ticket, which would be introduced via an intermediate dependency. But a gem author probably wouldn't even notice the windows-specific gem, in which case...

Someone using this gem that has ordinary ruby-vips as a intermediate dependency, on windows... would not get the special windows dependency. Which might mean things wouldn't work for them, unless they knew how to take care of it (say, by adding the windows version of the dependency to their app gemfile manually, after only getting the platform-independent one as an indirect dependency? Or just by manually installing windows libvips, that's what the special windows dependency is doing right?).

Seems confusing to me. Whether it's worth the confusion I couldn't say, I still don't totally understand the problem case (or how these windows-only gem metadata dependencies work. I wonder if we can find anyone who is an expert in how these windows metadata dependencies work, to suggest alternate solutions?).