Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add fetchLDM() - access Kellogg Soil Survey Lab Data Mart via Soil Data Access or local SQLite snapshot #243

Merged
merged 31 commits into from
Apr 8, 2022
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
b49f413
Add "chunking" for horizon data queries
brownag May 11, 2021
640d1cd
Hide SDA_query warnings and handle try-error results
brownag May 11, 2021
e24010d
Allow limited use of area_* columns for SSA
brownag May 13, 2021
5e05e55
Add `tables` for join of lab_layer to other tables
brownag May 13, 2021
750184e
More intelligent about chunk.size too big
brownag May 13, 2021
272baf4
fetchLDM: \donttest examples
brownag May 14, 2021
38b3314
fetchLDM: ntries argument, ensure chunk.size >= 1
brownag May 14, 2021
a6536e3
Docs
brownag May 14, 2021
01687c0
fetchLDM: "fixes" for schema differences in SQLite v.s. SDA; case-ins…
brownag May 14, 2021
e5cf8dd
Notes regarding many:1 relationships between lab_layer and analytes
brownag May 14, 2021
93e84ec
fetchLDM: tests
brownag May 18, 2021
3fcc263
Add filtering for prep_code and analyzed_size_frac (handles common SP…
brownag Jun 14, 2021
0291cd7
Update tests + docs
brownag Jun 15, 2021
619fe34
Handle NULL prep_code
brownag Jun 17, 2021
8d1a0fb
Add layer_type filtering argument for horizon, layer, reporting layer…
brownag Jun 17, 2021
91b0a24
Set default values for arguments, allow SSA area in what=... for SDA
brownag Jun 17, 2021
e01e1e3
Allow querying whole db with x=NULL (default)
brownag Jul 22, 2021
70f505b
Docs
brownag Feb 10, 2022
c023bf5
Make `what` argument more generic
brownag Feb 17, 2022
ca0c880
Merge branch 'master' into fetchLDM
brownag Feb 17, 2022
1083b05
Default `what` is pedlabsampnum
brownag Feb 17, 2022
daad24d
allow querying only fractionated tables lab_mineralogy_glass_countm a…
brownag Feb 19, 2022
c81c8b2
Docs
brownag Feb 19, 2022
deca372
Update for MIR SQLite DB
brownag Mar 4, 2022
050f610
Fix chunking
brownag Mar 10, 2022
a35b44e
Merge branch 'master' into fetchLDM
brownag Mar 14, 2022
930efeb
Merge branch 'master' into fetchLDM
brownag Mar 14, 2022
8801fd4
Merge branch 'master' into fetchLDM
brownag Mar 14, 2022
210b658
Merge branch 'master' into fetchLDM
brownag Apr 6, 2022
d3aecd4
fetchLDM: Put `dsn` argument at end / docs
brownag Apr 7, 2022
2788600
Add `lab_mir` to docs/options for tables
brownag Apr 7, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Allow querying whole db with x=NULL (default)
  • Loading branch information
brownag committed Feb 10, 2022
commit e01e1e310aebed27d40fd7d8f110972cc1121c59
25 changes: 12 additions & 13 deletions R/fetchLDM.R
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
#'
#' LDM model diagram: \url{https://jneme910.github.io/Lab_Data_Mart_Documentation/Documents/SDA_KSSL_Data_model.html}
#'
#' @param x a vector of values to find in column specified by `what`
#' @param x a vector of values to find in column specified by `what`, default `NULL` uses no constraints on `what`
#' @param dsn data source name; either a path to a SQLite database, an open DBIConnection or (default) `NULL` (to use `soilDB::SDA_query`)
#' @param what a single column name from tables: `lab_combine_nasis_ncss`, `lab_webmap`, `lab_site`, `lab_pedon` or `lab_area`
#' @param bycol a single column name from `lab_layer` used for processing chunks; default: `"pedon_key"`
Expand Down Expand Up @@ -45,7 +45,7 @@
#' }
#' @importFrom aqp `depths<-` `site<-`
#' @importFrom data.table rbindlist
fetchLDM <- function(x,
fetchLDM <- function(x = NULL,
dsn = NULL,
what = "lab_combine_nasis_ncss.pedlabsampnum",
bycol = "pedon_key",
Expand Down Expand Up @@ -77,7 +77,8 @@ fetchLDM <- function(x,
con <- NULL
}

what <- match.arg(what, choices = c("pedon_key", "site_key", "pedlabsampnum", "pedoniid", "upedonid",
if (!is.null(x)) {
what <- match.arg(what, choices = c("pedon_key", "site_key", "pedlabsampnum", "pedoniid", "upedonid",
"labdatadescflag", "priority", "priority2", "samp_name", "samp_class_type",
"samp_classdate", "samp_classification_name", "samp_taxorder",
"samp_taxsuborder", "samp_taxgrtgroup", "samp_taxsubgrp", "samp_taxpartsize",
Expand All @@ -99,21 +100,20 @@ fetchLDM <- function(x,
"corr_taxfamhahatmatcl", "SSL_taxfamhahatmatcl", "pedobjupdate",
"siteobjupdate", "area_key", "area_type", "area_sub_type", "parent_area_key",
"parent_org_key", "area_code", "area_name", "area_abbrev", "area_desc"))

}
# TODO: set up arbitrary area queries by putting area table into groups:
# country, state, county, mlra, ssa, npark, nforest

# get site/pedon/area information
site_query <- sprintf(
site_query <- paste0(
"SELECT * FROM lab_combine_nasis_ncss
LEFT JOIN lab_webmap ON
lab_combine_nasis_ncss.pedon_key = lab_webmap.pedon_key
LEFT JOIN lab_site ON
lab_combine_nasis_ncss.site_key = lab_site.site_key
LEFT JOIN lab_pedon ON
lab_combine_nasis_ncss.site_key = lab_pedon.site_key
WHERE LOWER(%s) IN %s",
what, format_SQL_in_statement(tolower(x)))
lab_combine_nasis_ncss.site_key = lab_pedon.site_key ",
ifelse(is.null(x), "", sprintf("WHERE LOWER(%s) IN %s", what, format_SQL_in_statement(tolower(x)))))

if(inherits(con, 'DBIConnection')) {
# query con using (modified) site_query
Expand Down Expand Up @@ -269,11 +269,10 @@ fetchLDM <- function(x,
layer_type <- match.arg(layer_type, c("horizon", "layer", "reporting layer"), several.ok = TRUE)

layer_query <- sprintf(
"SELECT * FROM lab_layer %s WHERE lab_layer.layer_type IN %s AND %s IN %s AND %s",
"SELECT * FROM lab_layer %s WHERE lab_layer.layer_type IN %s %s AND %s",
paste0(sapply(flattables[flattables %in% tables], function(a) tablejoincriteria[[a]]), collapse = "\n"),
format_SQL_in_statement(layer_type),
bycol,
format_SQL_in_statement(x),
ifelse(is.null(x), "", paste0(" AND ", bycol, " IN ", format_SQL_in_statement(x))),
paste0(paste0(sapply(flattables[flattables %in% tables[tables != "lab_rosetta_Key"]],
function(b) paste0("IsNull(",b,".prep_code, '')")),
" IN ", format_SQL_in_statement(prep_code)), collapse= " AND "))
Expand All @@ -296,8 +295,8 @@ fetchLDM <- function(x,
}

if(inherits(con, 'DBIConnection')) {
# query con using (modified) layer_query
return(try(DBI::dbGetQuery(con, gsub("\\blab_|\\blab_combine_|_properties|_Key|_including_estimates_and_default_values|_and|mineralogy_|_count", "", gsub("major_and_trace_elements_and_oxides","geochemical",layer_query)))))
# query con using (modified) layer_query for SQLite
return(try(DBI::dbGetQuery(con, gsub("IsNull", "IFNULL", gsub("\\blab_|\\blab_combine_|_properties|_Key|_including_estimates_and_default_values|_and|mineralogy_|_count", "", gsub("major_and_trace_elements_and_oxides","geochemical", layer_query))))))
}

layerdata <- suppressWarnings(SDA_query(layer_query))
Expand Down
4 changes: 2 additions & 2 deletions man/fetchLDM.Rd

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.