Add new page URLs during import

This is pretty conservative -- we don't touch redirect URLs since it's not uncommon for a page to redirect somewhere else (usually a parent path) instead of sending a 403/404 when it gets removed. I left a TODO about the situation to prompt further ideation on strategies to do better.

Author: Mr0grog

app/jobs/import_versions_job.rb

@@ -191,6 +191,20 @@ def page_for_record(record, create: true, row:)
     (record['page_maintainers'] || []).each {|name| page.add_maintainer(name)}
     (record['page_tags'] || []).each {|name| page.add_tag(name)}
 
+    # If the page was not an *exact* URL match, add the URL to the page.
+    # (`page.find_by_url` used above will match by `url_key`, too.)
+    page.urls.find_or_create_by(url: url)
+    # TODO: Add URLs from redirects automatically. The main blocker for
+    # this at the moment is the following situation:
+    #
+    #   Two pages, A=https://example.com/about
+    #              B=https://example.com/about/locations
+    #   Page B is removed, but instead of returning a 404 or 403 status
+    #     code, it starts redirecting to Page A.
+    #
+    # This is unfortunately not uncommon, so we need some heuristics to
+    # account for it, e.g. URL does not already belong to another page.
+
     page
   end
 

test/jobs/import_versions_job_test.rb

@@ -318,4 +318,27 @@ def teardown
     version = pages(:home_page).latest
     assert_equal('text/html', version.media_type)
   end
+
+  test 'adds URL to an existing page if the version was matched to a page with a different URL' do
+    page = Page.create(url: 'https://example.gov/office')
+    import = Import.create_with_data(
+      { user: users(:alice) },
+      [
+        {
+          page_url: 'http://example.gov/office/',
+          capture_time: Time.now - 1.second,
+          uri: 'https://test-bucket.s3.amazonaws.com/whatever',
+          version_hash: 'abc',
+        }
+      ].map(&:to_json).join("\n")
+    )
+    ImportVersionsJob.perform_now(import)
+
+    assert_equal(1, page.versions.count, 'Version was added to the right page')
+    assert_equal(
+      ['https://example.gov/office', 'http://example.gov/office/'],
+      page.urls.pluck(:url),
+      'New URL was added to the page'
+    )
+  end
 end