Changing keepDate to allow multiple dates, would close #108

[ianmilligan1]

GitHub issue(s):

If you are responding to an issue, please mention their numbers below.

• KeepDate UDF should support date range #108

What does this Pull Request do?

This Pull Request changes the use of keepDate. Right now, it is used like:

  .keepDate("2008", YYYY)

If a user wanted data from 2008 and 2010, they would have to run two separate scripts - one for 2008 and one for 2010.

This new format allows the use of a list in keepDate, so a command like:

  .keepDate(List("2007","2010"), YYYY)

How should this be tested?

I have tested this on a collection of WARCs with files from 2007 and from 2010. Running:

import io.archivesunleashed.spark.matchbox.RecordLoader
import io.archivesunleashed.spark.rdd.RecordRDD._
import io.archivesunleashed.spark.matchbox.{RemoveHTML, RecordLoader}
import io.archivesunleashed.spark.matchbox.ExtractDate.DateComponent._

RecordLoader.loadArchives("/Users/ianmilligan1/dropbox/testdata/*.gz", sc)
  .keepValidPages()
  .keepDate(List("2007","2010"), YYYY)
  .map(r => (r.getCrawlDate, r.getDomain, r.getUrl, RemoveHTML(r.getContentString)))
  .saveAsTextFile("/users/ianmilligan1/desktop/output/2007and2010")

Resulted in all the records from 2007 and 2010. You can also still test only one year, but it still needs to be provided as a list.

Additional Notes:

If approved, this will require revision to all keepDate documentation.

Interested parties

@ruebot @greebie @lintool

[ianmilligan1]

Hmm, the code coverage is slightly lower, but so far as I can tell, that line was never covered before. Maybe just because it is longer now?

[codecov]

Codecov Report

Merging #161 into master will not change coverage.
The diff coverage is 50%.

@@           Coverage Diff           @@
##           master     #161   +/-   ##
=======================================
  Coverage   67.95%   67.95%           
=======================================
  Files          38       38           
  Lines         774      774           
  Branches      143      142    -1     
=======================================
  Hits          526      526           
  Misses        199      199           
  Partials       49       49

Impacted Files	Coverage Δ
...ala/io/archivesunleashed/spark/rdd/RecordRDD.scala	70.37% <50%> (ø)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 128b86d...c81fde1. Read the comment docs.

[greebie]

Yes. If you add content to an uncovered line, it will decrease overall coverage.

[ruebot]

In your example you use two dates, you should use two dates here as well.

[ianmilligan1]

Added in latest commit, thanks for the review!