Handle wget WARC-Target-URI formatting.

[ruebot]

GitHub issue(s): #514

Handle wget WARC-Target-URI formatting.

What does this Pull Request do?

• Resolves WARC-Target-URI in Wget warc files is not parsed properly #514
• Regex replacement for getUrl
• Add test
• Add test fixture

How should this be tested?

Test should take care of it. Though, if you want to test locally the following Scala and Python examples should do it:

import io.archivesunleashed._
import io.archivesunleashed.udfs._

RecordLoader.loadArchives("/home/nruest/Projects/au/aut/src/test/resources/warc/issue-514.warc", sc).all()
  .select($"crawl_date", $"url")
  .show(2, false)

from aut import *

WebArchive(sc, sqlContext, "/home/nruest/Projects/au/aut/src/test/resources/warc/issue-514.warc") \
  .all() \
  .select("crawl_date", "url") \
  .show(2, False)

Both should produce the following:

+----------+-----------------------------+
|crawl_date|url                          |
+----------+-----------------------------+
|20210511  |http://www.archiveteam.org/  |
|20210511  |https://wiki.archiveteam.org/|
+----------+-----------------------------+

Interested parties

@javieraespinosa let me know if this should resolve your issue. If it does, let me know so we can merge, and I'll cut a release.

[codecov]

Codecov Report

Merging #515 (2d6b2e0) into main (8104a65) will increase coverage by 0.08%.
The diff coverage is 100.00%.

@@             Coverage Diff              @@
##               main     #515      +/-   ##
============================================
+ Coverage     88.83%   88.92%   +0.08%     
  Complexity       57       57              
============================================
  Files            43       43              
  Lines          1012     1020       +8     
  Branches         85       85              
============================================
+ Hits            899      907       +8     
  Misses           74       74              
  Partials         39       39              

[javieraespinosa]

Thanks @ruebot! Works like a charm.