Add last-modified http header.

[ruebot]

Right now this is just the string from the headers, and not converted to a date format. Not sure if we should leave it as is, or convert it to a format like we do for crawl_date.

scala> RecordLoader.loadArchives(data, sc).all().select($"crawl_date", $"last_modified").show(10, false)
[2022-10-07T18:40:00.043Z - 00002 - HdfsIO] Opening file file:/home/nruest/Projects/au/sample-data/geocities/GEOCITIES-20091027143300-00114-ia400112.us.archive.org.warc.gz (Offset: 0, length: 0, decompress: false, strategy: BlockWise [dynamic])
+--------------+-----------------------------+
|crawl_date    |last_modified                |
+--------------+-----------------------------+
|20091027143300|                             |
|20091027143259|Sat, 23 Sep 2000 23:34:54 GMT|
|20091027143259|Fri, 13 Sep 2002 16:30:29 GMT|
|20091027143300|Mon, 11 Feb 2002 15:45:53 GMT|
|20091027143259|Sat, 19 Sep 1998 16:47:03 GMT|
|20091027143259|Fri, 25 Jan 2008 15:03:03 GMT|
|20091027143300|Fri, 21 Sep 2001 22:46:58 GMT|
|20091027143258|Thu, 09 Oct 2008 01:52:03 GMT|
|20091027143300|                             |
|20091027143259|Tue, 16 Apr 2002 14:51:03 GMT|
+--------------+-----------------------------+
only showing top 10 rows

[ruebot]

Gonna close this, and leave an example on how to implement it for any arbitrary http header in case it's useful for anybody else in the future.

Thanks to @helgeho for pointing out that it's possible to just use headerMap 😃

Example I'm using in aut:

def getLastModified: String =
    http(warc)
      .flatMap(_.headerMap.get("last-modified"))
      .getOrElse("")