Hathi EF (Exracted Features) data

[sckott]

via benjamin schmidt https://twitter.com/benmschmidt/status/1062443526275977216

resources:

• https://analytics.hathitrust.org/datasets
• https://wiki.htrc.illinois.edu/display/COM/Extracted+Features+Dataset#ExtractedFeaturesDataset-getdataEFv.2
• the python lib https://github.com/htrc/htrc-feature-reader

---

list files (but this is 1.2 gigs, so warn users it will take a while)

syncr::rsync(src = "data.analytics.hathitrust.org::features/listing/htrc-ef-all-files.txt", dest = "htrc-ef-all-files.txt")

get a file:

syncr::rsync(src = "data.analytics.hathitrust.org::features/mdp/pairtree_root/39/07/60/00/48/48/11/39076000484811/mdp.39076000484811.json.bz2", dest = "mdp.39076000484811.json.b
    z2")

[sckott]

needs https://github.com/mrc-ide/syncr

[bmschmidt]

Thanks! Would love to see this.

From using the python module, I find this to be especially useful with two features:

1. The ability to go from an htid to the straightforward pairtree link (which is pretty straightforward), which would integrate well with the existing metadata calls.

2. Maintaining a local on-disk library, so that you generally load from local files, but the library can fall back to fetching from Hathi if you request a remote file.

[bmschmidt]

Regardless of how you get there, there also needs to be data pulled out it! Here's some code that builds out the most useful table you can get from EF: a wordcount list for the entire book.

response: a tibble with 45,000 rows, columns for page, section of page, count, and part of speech.

   section word        page POS   count
   <chr>   <chr>      <int> <chr> <int>
 1 body    management   167 NN       11
 2 body    newspapers   182 NNS      10
 3 body    children      55 NNS       9
 4 body    children     174 NNS       9
 5 body    companies    141 NNS       9
 6 body    companies    143 NNS       9
 7 body    favorable     29 JJ        9
 8 body    favorable     33 JJ        9
 9 body    favorable     69 JJ        9
10 body    newspapers   181 NNS       9
# ... with 9,663 more rows

My first pass at code to do this is very slow; ~5 seconds for a book. I don't know the fastest way to do this kind of list manipulation in R.

listified_version = jsonlite::read_json("mdp.39076000484811.json.bz2")

data_frame(page = 1:length(listified_version$features$pages)) %>% 
    group_by(page) %>% 
    do(list_to_df(listified_version$features$pages[[.$page[[1]]]], "section")) %>% 
    filter(section %in% c("footer", "header", "body")) %>% 
    group_by(section, page) %>% 
    do({list_to_df(.$l[[1]]$tokenPosCount, "word")}) %>% 
    group_by(section, word, page) %>% 
    do(list_to_df(.$l[[1]], "POS")) %>% 
    mutate(count = unlist(l)) %>% 
    select(-l)

list_to_df <- function(listfordf, name = "name"){
  # Hadley Wickham's function from https://gist.github.com/aammd/9ae2f5cce9afd799bafb
  #; should be fast from the comments, though I find it takes maybe a second per book.

  if(!is.list(listfordf)) stop("it should be a list")

  df <- list(l = listfordf)
  class(df) <- c("tbl_df", "data.frame")
  attr(df, "row.names") <- .set_row_names(length(listfordf))

  if (!is.null(names(listfordf))) {
    df[[name]] <- names(listfordf)
  }

  df
}

[sckott]

thanks @bmschmidt !

wrt 1 and 2.

1. sounds good, can do
2. by "Maintaining a local on-disk library" - what do you mean exactly? the big metadata file of file names? or just everything in general?

---

Thanks for the code and example data.frame. The code works for me. Can try to work on the speed.

[bmschmidt]

"Local on-disk library..."

Usually when I work with Hathi Data, I'll have a folder in the directory called 'hathi-ef' or something to store the bzip files for each individual book; I don't want to keep downloading them again and again whenever I run the script.
So if I call extended_features(htid) (or whatever the function is) I want it to first check that local stash, and then (unless told not to) pull the documents down from hathi into the local stash.

Since EF uses rsync, it's about as convenient to keep this organized by the pairtree structure, and if you get above a few thousand files can save some directory organization problems.

This is sort of a matter of personal style, though; downloading to a tmp location is fine to. In my imagination you'd use this to work with a corpus as often as individual books, but I dunno!

[sckott]

thanks. agree on this. was just checking to make sure.

[bmschmidt]

I've been looking into this again for a class I'm teaching, and got things working to my satisfaction. Want it cleaned up into a pull request?

Two advances over the above.

1. Hathi has added HTTP endpoints for uncompressed JSON files. This is nice; it means that rsyncr isn't required. Link
2. I started just using 'unlist' on the json features and then cleaning them up, which works perfectly well. It's not blazingly fast, but it seems to beat descending the tree manually.

library(stringr)
library(tibble)
library(dplyr)
library(tidyr)


id_encode = function(htid) htid %>% id_clean %>% str_replace_all("\\.", ",")
id_clean = function(htid) htid %>% str_replace_all(":", "+") %>% str_replace_all("/", "=")

pairtree = function(htid) {
  splitted = str_split(htid, "\\.", n = 2)[[1]]
  breaks = seq(1, nchar(splitted[2]), by = 2)
  cleaned = splitted[2] %>% id_encode
  slashes = str_sub(cleaned, breaks, breaks + 1) %>%  str_c(sep="/", collapse="/")
  str_c(splitted[1], "pairtree_root", slashes, cleaned, sep="/") 
}

local_loc = function(htid, dir = "features") {
  clean = htid %>% id_clean
  pairtr = pairtree(htid)
  str_glue("{dir}/{pairtr}/{clean}.json.bz2")
}

download_http = function(htid, dir = "features") {
  pairtr = pairtree(htid)
  clean = htid %>% id_clean
  url = str_glue("https://data.analytics.hathitrust.org/htrc-ef-access/get?action=download-ids&id={htid}&output=json")
  local_name = local_loc(htid)
  if (!file.exists(local_name)) {
    dir.create(dirname(local_name), showWarnings = FALSE, recursive = TRUE)
    download.file(url, local_name)
  }

}

parse_page = function(page) {
  all_data = unlist(page)
  data_frame(
    seq = all_data[1], 
    count = as.numeric(all_data), 
    label = names(all_data)) %>% 
    # Split up the label: only the metadata pieces that actually give tokens will have four elements.
    separate(label, c("section", "p", "token", "POS"), fill = "left", extra="drop") %>% 
    filter(!is.na(section)) %>% # Drop non-token-count pieces.
    select(-p) # Dummy row--remove.
}

parse_listified_book = function(listified_version) {
  lapply(listified_version$features$pages, parse_page) %>% 
    bind_rows %>% mutate(page = as.numeric(seq)) %>% 
    select(-seq)
}

load_book = function(htid, dir = "features") {
  download_http(htid)
  jsonlite::read_json(local_loc(htid)) %>% parse_listified_book
}

library(ggplot2)
htid = "mdp.39076002771645"
f = load_book(htid)
library(ggplot2)

f %>% 
  group_by(token) %>%
  filter(sum(count) > 75) %>% # Only common words
  mutate(wordTotal = sum(count)) %>% # Store total occurences 
  mutate(lower = tolower((token))) %>%
  group_by(lower) %>%
  mutate(formTotal = sum(count)) %>% # Count the non-case specific forms
  filter(str_length(token) > 2, str_detect(token, "^[A-Z][a-z]")) %>% # Only capitalized words
  filter(wordTotal/formTotal > .95) %>% # Only words that appear overwhelmingly in the capped form
  ggplot() + geom_point(aes(x=page, size = count, y=reorder(token, count, FUN = sum), color=token))

[sckott]

nice @bmschmidt !

Tried it out, works well.

Do you want to have this functionality in this pkg? Or are you happy with your script as standalone?

[sckott]

Any thoughts on my comment @bmschmidt ?

[bmschmidt]

Sorry for leaving this dangling! I ended up just fixing up a few problems with these functions and bundling them together into a standalone package for my class. The functionality is pretty different (no shared imports), so I don't know if the usefulness of the shared subject justifies all the extra imports, etc. I've added on.

Ben

[sckott]

Maybe it still makes sense to do the data download part in this pkg? or no?

[bmschmidt]

So data download over http here, but more opinionated parsing of the JSON to tibbles elsewhere? That would make sense to me, so this package would still be a standard wrapper around hathi's various APIs. I suspect all the pairtree/file location infrastructure can be dropped from this package.

[sckott]

Okay, thanks, makes sense