Update Gem version requirements for some older libraries

[nightpool]

What does this PR do?

This PR updates the Gem version requirements for a few different libraries we currently use in production. I've tested these integrations locally and they all seem to work, and I've reviewed the changes between the versions currently required and the older versions, and found no major changes.

Motivation

As part of our Datadog integration for Genius, we went through the list of incompatible Gems detected by auto-instrumentation. I identified these gems as having very minor changes between the currently-incompatible versions we use and the versions required by Datadog, enabling use them via autoinstrumentation.

Additional Notes

I have not tested these changes in production but I have tested them locally and on our staging environment.

I understand if Datadog feels these changes are too risky or represent too much of a breaking change to accept. Open to suggestions on how we can enable these integrations without maintaining our own fork of the ddtracerb gem.

How to test the change?

In theory, the best way to test these changes is to update whatever integration tests already exist for these libraries with the older versions. I haven't looked into that.

[TonyCTHsu]

👋 @nightpool , thanks for submitting this. I feel your pain to maintain something old and legacy and I will bring this to our team for discussion.

I am kinda interested to explore some alternatives. For instance, introducing a force option to the integration configurations. The force option would force applying patches without checking versions, even the version could be out of scope of our support, but user has to verify the behaviour themselves. What do you think?

[TonyCTHsu]

Or I think i would be easier for you to patch #patchable? , something like

module Datadog
  module Tracing
    module Contrib
      module Faraday
        # Description of Faraday integration
        class Integration
          def self.patchable?
            true
          end
        end
      end
    end
  end
end

[nightpool]

I don't hate that idea. One issue with it is that we're currently using the require: 'ddtrace/auto_instrument' helper, which means that we don't have the opportunity to do any patching prior to the initial instrumentation load.

This is valuable to us because it means we don't have to specify every specific gem we're using manually (we're using at least 6 HTTP gems in different parts of the code base, for example....). Is there a way to get this "auto-instrument" functionality but at a later point in the Rails loading process (after we've had the opportunity to configure Datadog?).

[TonyCTHsu]

👋 @nightpool , our auto-instrumentation would hook into rails lifecycle with our default configuration and version checks, which can still be overwriten by config/initializers/datadog.rb.

I would recommend you to manually patching them in config/initializers/datadog.rb.

Datadog::Tracing::Contrib::Faraday::Patcher.patch
Datadog::Tracing::Contrib::Dalli::Patcher.patch
...

This would skip the version check, would that work for you?

[nightpool]

@TonyCTHsu One issue with that approach is that it leads to a lot of misleading log lines every time e.g. a developer starts a console:

Loading development environment (Rails 2.3.18)
Hashie skipping railtie as cannot load such file -- rails/railtie
[ddtrace] Unable to patch Datadog::Tracing::Contrib::ActionMailer::Integration (Available?: true, Loaded? true, Compatible? false, Patchable? false)
[ddtrace] Unable to patch Datadog::Tracing::Contrib::ActionView::Integration (Available?: true, Loaded? true, Compatible? false, Patchable? false)
[ddtrace] Unable to patch Datadog::Tracing::Contrib::ActiveRecord::Integration (Available?: true, Loaded? true, Compatible? false, Patchable? false)
[ddtrace] Unable to patch Datadog::Tracing::Contrib::ActiveSupport::Integration (Available?: true, Loaded? true, Compatible? false, Patchable? false)
[ddtrace] Unable to patch Datadog::Tracing::Contrib::Rails::Integration (Available?: true, Loaded? true, Compatible? false, Patchable? false)
[1] dev (main)> 

As you can see, we already get a significant number of those log-lines for gems we have but we know are incompatible. Adding log lines for compatible gems that are just loaded differently would make the situation much worse. Is there any way to silence these log lines for individual gems (but still get notifications/auto-instrumentation for new gems)? Otherwise we may just move away from auto-instrumentation entirely.

[nightpool]

Also, just overall, the changes in the versions mentioned in this PR are very very small (with the exception of rake).

[nightpool]

Thanks all for the feedback. This is my new datadog.rb initializer file after the changes, LMK what you think

if AppConfig.datadog_enabled?
  if RAILS_2
    [
      Datadog::Tracing::Contrib::ActionMailer::Integration,
      Datadog::Tracing::Contrib::ActionView::Integration,
      Datadog::Tracing::Contrib::ActiveRecord::Integration,
      Datadog::Tracing::Contrib::ActiveSupport::Integration,
      Datadog::Tracing::Contrib::Rails::Integration
    ].each do |integration_class|
      integration_class.prepend(Module.new do
        def auto_instrument?
          false
        end
      end)
    end
  end

  {
    Dalli: ['2.0.0', '1.0.0'],
    Ethon: ['0.11.0', '0.9.0'],
    Faraday: ['0.14.0', '0.9.0'],
    Rake: ['12.0', '10.0']
  }.each do |(integration, (old_minimum_version, new_minimum_version))|
    klass = Datadog::Tracing::Contrib.const_get(integration).const_get('Integration')
    found_version_requirement = klass.send(:remove_const, 'MINIMUM_VERSION')
    raise "Expected Datadog integration version requirement #{found_version_requirement} to match #{old_minimum_version}" unless found_version_requirement.to_s == old_minimum_version
    klass.const_set('MINIMUM_VERSION', Gem::Version.new(new_minimum_version))
  end

  require 'ddtrace/auto_instrument'

  Datadog.configure do |c|
    c.env = Rails.env
    c.service = 'MY_SERVICE_NAME'
    c.tags = { rails_version: Rails::VERSION::MAJOR.to_s }
    c.profiling.enabled = true

    c.tracing.instrument :rails, {middleware_names: true} if RAILS_3
    c.tracing.instrument :resque, {service_name: 'resque'}
  end

  if RAILS_2
    Rails.configuration.middleware.insert(0, Datadog::Tracing::Contrib::Rack::TraceMiddleware)
  end
end

[marcotc]

This is great work, @nightpool! Thanks for testing it as well.

Could you also update the minimum version in the compatibility table? https://github.com/DataDog/dd-trace-rb/blob/master/docs/GettingStarted.md#integration-instrumentation