Created the new CIRecordHelper behaviour and an associate rake task:

- Recording the deprecations whitelist for thousands of tests is a
  real pain depending on how your CI is setup.

  Most CI runs test in parallel in different machines, the regular
  `Record` Behaviour from this gem will create bunch of YAML files,
  but since multiple test from the same test file runs on different
  machines, you end up with multiple yaml files for the same test
  suite. You'd need a way to download all artifacts from CI and
  "merge" all files, it's a nightmare.

  This PR introduce a new behaviour that leverage a feature that most
  CI have which is to download the log output from the test run.
  The `CIRecordHelper` behaviour will output a JSON representation of
  your deprecations. You'll simply have to download the log output
  locally and finally run the associate rake task meant to parse this
  log file and generate proper deprecation as YAML file.

README.md

@@ -52,6 +52,7 @@ This gem provides 3 behaviors, the default one being `DeprecationToolkit::Behavi
   - `DeprecationToolkit::DeprecationIntroduced` error if a new deprecation is introduced.
   - `DeprecationToolkit::DeprecationRemoved` error if a deprecation was removed (compare to the one recorded in the shitlist).
 * `DeprecationToolkit::Behaviors::Record` will record deprecations.
+* `DeprecationToolkit::Behaviors::CIRecordHelper` See separated explanation below.
 * `DeprecationToolkit::Behaviors::Disabled` will do nothing.
   - This is useful if you want to disable this gem for a moment without removing the gem from your Gemfile.
 
@@ -71,6 +72,15 @@ end
 DeprecationToolkit::Configuration.behavior = StatsdBehavior
 ```
 
+##### DeprecationToolkit::Behaviors::CIRecordHelper
+
+This is a special type of behaviour meant to help you record deprecations if you have a lof of them.
+Imagine if you have thousands of tests and need to record deprecations for each on your machine, this is going to take ages.
+Recording deprecations on CI with the regular `Record` behavior isn't possible because of the way CI parallelize test in multiple container (Multiple tests from the same file runs in parallel in diferrent machines, the deprecation files that get created are then splitted. Regrouping them is a nightmare.)
+
+This behaviour will output a JSON representation of your deprecations. Your CI should have a way to download the log generated from the test run. Download it on your locale machine. Finally run the rake task `FILEPATH=<path_to_downloaded_log> deprecation_toolkit:record_from_ci_output`.
+This task will parse your CI log and grab the output generated by the DeprecationToolkit and will finally convert everything back to YML files.
+
 ### 🔨 `#DeprecationToolkit::Configuration#allowed_deprecations`
 
 You can ignore some deprecations using `allowed_deprecations`. `allowed_deprecations` accepts an array of Regexp and Procs.

lib/deprecation_toolkit.rb

@@ -1,5 +1,7 @@
 # frozen_string_literal: true
 
+load 'tasks/ci_recorder.rake'
+
 module DeprecationToolkit
   autoload :DeprecationSubscriber,     "deprecation_toolkit/deprecation_subscriber"
   autoload :Configuration,             "deprecation_toolkit/configuration"
@@ -10,6 +12,7 @@ module Behaviors
     autoload :Disabled,                "deprecation_toolkit/behaviors/disabled"
     autoload :Raise,                   "deprecation_toolkit/behaviors/raise"
     autoload :Record,                  "deprecation_toolkit/behaviors/record"
+    autoload :CIRecordHelper,          "deprecation_toolkit/behaviors/ci_record_helper"
   end
 
   def self.add_notify_behavior

lib/deprecation_toolkit/behaviors/ci_record_helper.rb

@@ -0,0 +1,25 @@
+# frozen_string_literal: true
+
+require 'json'
+
+module DeprecationToolkit
+  module Behaviors
+    class CIRecordHelper
+      extend ReadWriteHelper
+
+      HEADER = '[DeprecationToolkit]'
+
+      def self.trigger(test, current_deprecations, recorded_deprecations)
+        filename = recorded_deprecations_path(test)
+
+        to_output = {
+          filename.to_s => {
+            test.name => current_deprecations.deprecations_without_stacktrace
+          }
+        }
+
+        raise "#{HEADER} #{JSON.dump(to_output)}"
+      end
+    end
+  end
+end

lib/deprecation_toolkit/behaviors/record.rb

@@ -6,7 +6,9 @@ class Record
       extend ReadWriteHelper
 
       def self.trigger(test, collector, _)
-        write(test, collector.deprecations_without_stacktrace)
+        deprecation_file = recorded_deprecations_path(test)
+
+        write(deprecation_file, { test.name => collector.deprecations_without_stacktrace })
       end
     end
   end

lib/deprecation_toolkit/read_write_helper.rb

@@ -7,21 +7,22 @@
 module DeprecationToolkit
   module ReadWriteHelper
     def read(test)
-      deprecation_file = recorded_deprecations_path(test)
+      deprecation_file = Bundler.root.join(recorded_deprecations_path(test))
       YAML.load(deprecation_file.read).fetch(test.name, [])
     rescue Errno::ENOENT
       []
     end
 
-    def write(test, deprecations)
-      deprecation_file = recorded_deprecations_path(test)
+    def write(deprecation_file, deprecations_to_record)
       create_deprecation_file(deprecation_file) unless deprecation_file.exist?
 
       content = YAML.load_file(deprecation_file)
-      if deprecations.any?
-        content[test.name] = deprecations
-      else
-        content.delete(test.name)
+      deprecations_to_record.each do |test_name, deprecations|
+        if deprecations.any?
+          content[test_name] = deprecations
+        else
+          content.delete(test_name)
+        end
       end
 
       if content.any?
@@ -45,7 +46,7 @@ def recorded_deprecations_path(test)
         Configuration.deprecation_path
       end
 
-      Bundler.root.join(deprecation_folder, "#{test.class.name.underscore}.yml")
+      Pathname(deprecation_folder).join("#{test.class.name.underscore}.yml")
     end
 
     def test_location(test)

lib/tasks/ci_recorder.rake

@@ -0,0 +1,53 @@
+# frozen_string_literal: true
+
+require 'tempfile'
+require 'json'
+require 'active_support/core_ext/hash'
+require 'rake'
+require_relative '../deprecation_toolkit/read_write_helper'
+
+class CIRecorder
+  include Rake::DSL
+  include DeprecationToolkit::ReadWriteHelper
+
+  def initialize
+    namespace :deprecation_toolkit do
+      desc 'Parse a file generated with the CIOutputHelper and generate deprecations out of it'
+      task :record_from_ci_output do
+        raw_file = ENV.fetch('FILEPATH')
+
+        deprecations = extract_deprecations_output(raw_file) do |file|
+          parse_file(file)
+        end
+
+        generate_deprecations_file(deprecations)
+      end
+    end
+  end
+
+  private
+
+  def extract_deprecations_output(file)
+    tmp_file = Tempfile.new
+    shell_command = "cat #{file} | sed -n -e 's/^.* \\[DeprecationToolkit\\] \\(.*\\)/\\1/p' > #{tmp_file.path}"
+
+    raise "Couldn't extract deprecations from output" unless system(shell_command)
+    yield(tmp_file)
+  ensure
+    tmp_file.delete
+  end
+
+  def parse_file(file)
+    file.each.with_object({}) do |line, hash|
+      hash.deep_merge!(JSON.parse(line))
+    end
+  end
+
+  def generate_deprecations_file(deprecations_to_record)
+    deprecations_to_record.each do |filename, deprecations|
+      write(Pathname(filename), deprecations)
+    end
+  end
+end
+
+CIRecorder.new

test/deprecation_toolkit/behaviors/ci_record_helper_test.rb

@@ -0,0 +1,76 @@
+# frozen_string_literal: true
+
+require "test_helper"
+require "tempfile"
+
+module DeprecationToolkit
+  module Behaviors
+    class CIRecordHelperTest < ActiveSupport::TestCase
+      setup do
+        @tempfolder = Dir.mktmpdir
+        @logfile_path = "#{@tempfolder}/ci_output.log"
+        @previous_configuration = Configuration.behavior
+        @previous_deprecation_path = Configuration.deprecation_path
+
+        Configuration.behavior = CIRecordHelper
+        Configuration.deprecation_path = @tempfolder
+      end
+
+      teardown do
+        Configuration.behavior = @previous_configuration
+        Configuration.deprecation_path = @previous_deprecation_path
+
+        FileUtils.rm_rf(@tempfolder)
+      end
+
+      test '#trigger raise an error whom message is parsable by the `deprecation_toolkit:record_from_ci_output` task' do
+        ci_output = +''
+        ci_output << "random stuff generated by CI, this should be filtered out by the Rake Task\n"
+
+        MyTest = Class.new(MiniTest::Test) do
+          def self.name
+            'MyTest'
+          end
+
+          def test_generate_deprecation
+            ActiveSupport::Deprecation.warn('Foo')
+          end
+
+          def test_generate_second_deprecation
+            ActiveSupport::Deprecation.warn('Bar')
+          end
+        end
+
+        generate_raw_failure_output(ci_output)
+        record_deprecation_files
+
+        success_result = MyTest.new('test_generate_deprecation').run
+        success_result2 = MyTest.new('test_generate_second_deprecation').run
+
+        assert_predicate success_result, :passed?
+        assert_predicate success_result2, :passed?
+      end
+
+      private
+
+      def generate_raw_failure_output(ci_output)
+        failure_result = MyTest.new('test_generate_deprecation').run
+        failure_result2 = MyTest.new('test_generate_second_deprecation').run
+
+        ci_output << failure_result.failures.first.message << "\n"
+        ci_output << failure_result2.failures.first.message << "\n"
+
+        File.write(@logfile_path, ci_output)
+      end
+
+      def record_deprecation_files
+        ENV["FILEPATH"] = @logfile_path
+        Rake::Task['deprecation_toolkit:record_from_ci_output'].invoke
+
+        assert File.exist?("#{@tempfolder}/my_test.yml")
+      ensure
+        ENV.delete("FILEPATH")
+      end
+    end
+  end
+end