Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
142 commits
Select commit Hold shift + click to select a range
08e51a0
added codecov to travis.yml
colinpmillar Aug 4, 2016
1ed472c
added covcov.yml
colinpmillar Aug 4, 2016
242775f
Added a first test
colinpmillar Aug 4, 2016
e99530b
added codecov to readme
colinpmillar Aug 4, 2016
aa042ac
Further work on covr
colinpmillar Aug 4, 2016
e032f71
Added covecov.yml to Rbuildignore
colinpmillar Aug 4, 2016
d95d0cd
updated readme [ci skip]
colinpmillar Aug 4, 2016
2d2b948
Added cran downloads [ci skip]
colinpmillar Aug 4, 2016
2653b3b
added license and link to gpl to readme [ci skip]
colinpmillar Aug 4, 2016
99aadca
update readme lisence [ci skip]
colinpmillar Aug 4, 2016
4d365bc
spelling [ci skip]
colinpmillar Aug 4, 2016
9c628ed
run travis build
slarge Aug 8, 2016
b332c5c
Fixes #1 bug with hh
colinpmillar Aug 9, 2016
94d3dff
Fixes #2 Remove trailing whitespace from HHdata columns
colinpmillar Aug 9, 2016
6c65268
Fix #3 Add getDATRAS function from rICES
colinpmillar Aug 9, 2016
9107e84
Update readme to include release version number [ci skip]
colinpmillar Aug 9, 2016
1e566cc
fixed bug if only one year supplied
colinpmillar Aug 9, 2016
062e072
Rearrange links to focus on DATRAS (icesDatras-package.Rd, README.md)
Aug 17, 2016
6801953
New function simplify() to format output data frame
Aug 17, 2016
37b4abc
Document changes and list authors for package version 1.1-0
Aug 17, 2016
ef68b27
Update (roxygenize) icesDatras-package.Rd
Aug 17, 2016
dccd64e
Update links on package help page
Aug 17, 2016
d4c9187
Remove white space with parseDatras()
Aug 19, 2016
af4ba78
export print method for DatrasDataOverview
colinpmillar Aug 19, 2016
4326744
Simplify example for getDatras
colinpmillar Aug 19, 2016
361c4f2
Apply consistent style (code+docs), no effect on functionality
Aug 19, 2016
f6e961b
Merge branch 'master' of github.com:ices-tools-prod/icesDatras
Aug 19, 2016
3d482b1
Add ellipsis to print.datrasoverview() to match generic print(x, ...)
Aug 20, 2016
2a1456a
Annotated function list on package help page
Aug 20, 2016
7dbffbe
Roxygenize package help page
Aug 20, 2016
d72a331
Apply consistent style (code+docs), no effect on functionality
Aug 20, 2016
6a970c0
Improve getDATRAS help page, linking to getHHdata, getHLdata, getCAdata
Aug 20, 2016
0094479
stiffle -> stifle -> suppress
Aug 21, 2016
afce847
Comment on trailing white space
Aug 21, 2016
0f04067
getSurveyList returns a vector, not data frame
Aug 22, 2016
f4b83f8
Convert all -9 values to NA
Aug 22, 2016
fbddbe8
Description: "R interface to access ..."
Aug 22, 2016
657a339
Whitespace
Aug 22, 2016
de62005
Whitespace
Aug 22, 2016
4db09b8
Shorten the 'see also' section of help pages
Aug 22, 2016
5394417
Correct spelling
slarge Aug 30, 2016
3a68929
Simplify URL link description
Aug 30, 2016
476d1b3
Add argument 'surveys' to getDatrasDataOverview()
Aug 30, 2016
e2006ab
Base examples on ROCKALL rather than NS-IBTS, to speed up package check
Aug 30, 2016
70513a2
Use year 2015 in Rockall examples
Aug 31, 2016
52608e7
Examples use Rockall 2015
Aug 31, 2016
7eb0952
Use Rockall 2012 in examples (fastest), enable all examples (no dontrun)
Aug 31, 2016
c9876b5
Simplify URL description
Aug 31, 2016
6227ace
Convert "" to NA, so for example hldata$Valid_Aphia is parsed as integer
Aug 31, 2016
4426184
parseDatras returns empty data frame if XML is empty, rather than crash
Sep 1, 2016
eee1090
Strip leading space as well
Sep 2, 2016
72bb4bb
Update NEWS
Sep 2, 2016
fab2801
simplified and speeded up XML parsing
colinpmillar Sep 14, 2016
1fd1e43
Add to last commit :|
colinpmillar Sep 14, 2016
f3e16eb
further improvements to XML parsing
colinpmillar Sep 14, 2016
5844c55
fixed small bug if only 1 column in XML output
colinpmillar Sep 14, 2016
9823dec
bug fix for empty XML objects being returned by webservice
colinpmillar Sep 14, 2016
6a471d1
added getCatchWgt function
colinpmillar Sep 14, 2016
4d1c4b8
added aphia code lookup table
colinpmillar Sep 14, 2016
eff8a54
trim white space from species names in aphia data
colinpmillar Sep 14, 2016
a4c91fc
added example using aphia data as lookup table to getCatchWgt
colinpmillar Sep 14, 2016
6d0e7f2
increase version number and added NEWS item
colinpmillar Sep 14, 2016
db70287
update readme to reflect new release
colinpmillar Sep 14, 2016
29321aa
Use replicate instead of sapply, since expression has no args
Sep 14, 2016
817947f
Elaborate comment on peek-pop algorithm
Sep 15, 2016
c3c4255
Update readme test
colinpmillar Sep 15, 2016
1bf5c62
remove R devel build test
colinpmillar Sep 15, 2016
8e90fd3
Update readme
colinpmillar Oct 6, 2016
0f984c0
Export the data checking funtions
colinpmillar Oct 24, 2016
6bccabc
aded space to readme sentence
colinpmillar Oct 24, 2016
28ae450
Simplify getDATRAS help page
Oct 28, 2016
4b61c5b
Cross reference help pages
Oct 29, 2016
9ae4135
Apply consistent style in help pages
Oct 29, 2016
5afca22
Uppercase Aphia and Latin
Oct 29, 2016
3fe5e79
Dummy declaration to appease roxygenize: aphia <- NULL
Oct 29, 2016
2592087
Roxygenize
Oct 29, 2016
75bc6d2
Replace "aphia" string with aphia <- NULL, to support plain roxygenize()
Nov 1, 2016
925fb47
trawl database -> trawl survey database
Nov 1, 2016
21f1be2
Linebreaks
Nov 1, 2016
e4a9e13
implements -> provides
Nov 1, 2016
b43be94
Remove aphia dataset (was halfway done in last commit)
Nov 1, 2016
0ca4c51
Apply common style to ICES package DESCRIPTION files
Nov 1, 2016
783851d
Remove knitr (not used)
Nov 1, 2016
d844605
Minor changes to comments
Nov 1, 2016
e2a1f19
Minor changes to comments
Nov 1, 2016
39fd8da
Remove link to ICES software applications
Nov 1, 2016
b864980
Documentation refers only to https://datras.ices.dk
Nov 1, 2016
ebd8c91
Documentation refers only to https://datras.ices.dk
Nov 1, 2016
f180f72
Documentation refers only to http://datras.ices.dk
Nov 1, 2016
c6405ad
Remove dot after URL
Nov 1, 2016
968968c
Whitespace
Nov 1, 2016
101adc2
Merge remote-tracking branch 'origin/master'
colinpmillar Nov 3, 2016
46d2798
added github icesVocab to travis
colinpmillar Nov 6, 2016
627aaed
fudge to close #10
colinpmillar Nov 7, 2016
008a70d
experimental parsing
colinpmillar Nov 7, 2016
c287636
fixed bug
colinpmillar Nov 7, 2016
83c2907
bug in experimental parsing
colinpmillar Nov 7, 2016
649cbb9
fixed bug in fudge!
colinpmillar Nov 7, 2016
3090fd1
hasty!
colinpmillar Nov 7, 2016
6defdbc
Revert "fixed bug in fudge!"
colinpmillar Nov 8, 2016
45b1b1f
Revert "bug in experimental parsing"
colinpmillar Nov 8, 2016
accae84
Revert "fixed bug"
colinpmillar Nov 8, 2016
5483e54
Revert "experimental parsing"
colinpmillar Nov 8, 2016
6898eef
fixed bug in fudge
colinpmillar Nov 8, 2016
f655d8a
Merge remote-tracking branch 'origin/master'
colinpmillar Nov 8, 2016
3992765
experimenting with direct parsing of XML
colinpmillar Nov 8, 2016
46d3f6e
Whitespace
Nov 8, 2016
3c736c7
Simplify simplify
Nov 8, 2016
aa1bd2d
Do not simplify StatRec column
Nov 8, 2016
6e07dbc
Simplify gsub(..., regmatches, gregexpr) to gsub(...)
Nov 8, 2016
84a93cf
Merge pull request #11 from arnima-github/direct_xml_parsing
colinpmillar Nov 8, 2016
0390ea1
Merge pull request #12 from ices-tools-prod/master
colinpmillar Nov 9, 2016
9cdb696
Comma in help page
Nov 10, 2016
77c00bd
numeric year -> year
Nov 10, 2016
3518408
Rearrange comments
Nov 12, 2016
c62328b
Merge pull request #13 from ices-tools-prod/master
colinpmillar Nov 28, 2016
826f445
Addind URL back into description as per CRAN requirements.
colinpmillar Dec 5, 2016
271ef10
Adding back in link to web service definition
colinpmillar Dec 5, 2016
7479d96
Merge remote-tracking branch 'refs/remotes/origin/master' into direct…
colinpmillar Dec 8, 2016
24d997c
Merge branch 'direct_xml_parsing' of https://github.com/ices-tools-pr…
colinpmillar Dec 8, 2016
1819a82
Merge remote-tracking branch 'refs/remotes/origin/direct_xml_parsing'…
colinpmillar Dec 8, 2016
ed429f9
update NEWS
colinpmillar Dec 8, 2016
a887834
update travis
colinpmillar Dec 8, 2016
e30fefc
updates to reading and parsing
colinpmillar Dec 9, 2016
25b8ff3
comment changes [ci skip]
colinpmillar Dec 9, 2016
20d8bb6
Short links
Dec 9, 2016
32d52c3
Short links
Dec 9, 2016
966f517
library(devtools), double quotes
Dec 9, 2016
4beb892
Simplify once
Dec 10, 2016
760fa59
Comments and whitespace
Dec 10, 2016
81580bc
return a more "tidy" nested list
slarge Dec 15, 2016
5146248
Merge pull request #14 from ices-tools-prod/slarge-patch-1
colinpmillar Jan 10, 2017
b924b08
Merge pull request #16 from ices-tools-prod/arni
colinpmillar Jan 10, 2017
5d66a5f
Merge branch 'master' into development
colinpmillar Jan 10, 2017
e52a01c
Merge remote-tracking branch 'refs/remotes/origin/master' into develo…
colinpmillar Jan 10, 2017
9005975
Merge branch 'development' of https://github.com/ices-tools-prod/ices…
colinpmillar Jan 10, 2017
2e4185e
Merge pull request #18 from ices-tools-prod/development
colinpmillar Jan 10, 2017
fb3188e
changes for release
colinpmillar Jan 10, 2017
d612dd5
Merge remote-tracking branch 'refs/remotes/origin/master' into develo…
colinpmillar Jan 10, 2017
60f6bcd
update travis - only r release on OSX
colinpmillar Jan 10, 2017
1469643
update comments [ci-skip]
colinpmillar Jan 10, 2017
23f16da
Merge pull request #20 from ices-tools-prod/development
colinpmillar Jan 10, 2017
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .Rbuildignore
Original file line number Diff line number Diff line change
Expand Up @@ -3,3 +3,5 @@ README.md
^.*\.Rproj$
^\.Rproj\.user$
^cran-comments\.md$
codecov.yml
^data-raw$
45 changes: 30 additions & 15 deletions .travis.yml
Original file line number Diff line number Diff line change
@@ -1,27 +1,42 @@
# .travis.yml for ices-tools packages

warnings_are_errors: false

os:
- linux

# Header
language: r
r:
- oldrel
- release
- devel

sudo: false
cache: packages
warnings_are_errors: false

# matrix: 3x Linux, 1x OS X
matrix:
include:
- r: release
- r: oldrel
- r: devel
- os: osx
osx_image: beta-xcode6.1
disable_homebrew: true
r: release
latex: false

# notifications
notifications:
email:
on_success: change
on_failure: change

repos:
CRAN: https://cloud.r-project.org

# r_packages
r_packages:
- icesVocab
- testthat
- rmarkdown

cache: packages
# r_github_packages
r_github_packages:
- jimhester/covr

notifications:
email:
on_success: change
on_failure: change
# after_success
after_success:
- Rscript -e 'covr::codecov()'
16 changes: 8 additions & 8 deletions DESCRIPTION
Original file line number Diff line number Diff line change
@@ -1,14 +1,14 @@
Package: icesDatras
Version: 1.0-0
Date: 2016-08-04
Version: 1.2-0
Date: 2017-01-10
Title: DATRAS Trawl Database Web Services
Authors@R: c(person("Colin", "Millar", role=c("aut","cre"), email="colin.millar@ices.dk"),
person("Einar", "Hjorleifsson", role="aut"))
Suggests: knitr
Imports: RCurl,
XML,
utils
Description: Functions that access the web services of the ICES (International
person("Scott", "Large", role="aut"),
person("Arni", "Magnusson", role="aut"))
Imports: utils
Suggests: icesVocab,
testthat
Description: R interface to access the web services of the ICES (International
Council for the Exploration of the Sea) DATRAS trawl survey
database <https://datras.ices.dk/WebServices/Webservices.aspx>.
License: GPL (>= 2)
Expand Down
11 changes: 7 additions & 4 deletions NAMESPACE
Original file line number Diff line number Diff line change
@@ -1,14 +1,17 @@
# Generated by roxygen2: do not edit by hand

S3method(print,datrasoverview)
export(checkSurveyOK)
export(checkSurveyYearOK)
export(checkSurveyYearQuarterOK)
export(getCAdata)
export(getCatchWgt)
export(getDATRAS)
export(getDatrasDataOverview)
export(getHHdata)
export(getHLdata)
export(getSurveyList)
export(getSurveyYearList)
export(getSurveyYearQuarterList)
importFrom(RCurl,basicTextGatherer)
importFrom(RCurl,curlPerform)
importFrom(XML,xmlParse)
importFrom(XML,xmlToDataFrame)
importFrom(utils,capture.output)
importFrom(utils,download.file)
45 changes: 45 additions & 0 deletions NEWS
Original file line number Diff line number Diff line change
@@ -1,3 +1,48 @@
--------------------------------------------------------------------------------
icesDatras 1.2-0 (2016-12-xx)
--------------------------------------------------------------------------------
o


--------------------------------------------------------------------------------
icesDatras 1.1-1 (2016-09-14)
--------------------------------------------------------------------------------
o Improved XML parsing, exploiting structure of data and that we can
delete XML nodes once read.

o Added function to extract catch weigthts by species and haul.



--------------------------------------------------------------------------------
icesDatras 1.1-0 (2016-08-17)
--------------------------------------------------------------------------------
o New argument 'surveys' in function getDatrasDataOverview() allows faster
overviews for one or few surveys.

o Improved XML parsing, so both leading and trailing white space is removed.

o Improved XML parsing, so -9 and "" is converted to NA.

o Improved XML parsing, so data frame columns are automatically coerced to the
appropriate storage mode (character, numeric, integer).




--------------------------------------------------------------------------------
icesDatras 1.0-3 (2016-08-09)
--------------------------------------------------------------------------------
o Fixed bug where code assumed stringsAsFactors = FALSE by default.

o Remove trailing white space from HHdata table entries.

o Added function getDATRAS() from retired package rICES to extract multiple
years and quarters.




--------------------------------------------------------------------------------
icesDatras 1.0-0 (2016-08-04)
--------------------------------------------------------------------------------
Expand Down
83 changes: 79 additions & 4 deletions R/checkInputs.R
Original file line number Diff line number Diff line change
@@ -1,5 +1,25 @@
#' @importFrom utils capture.output
#' Check that a survey name is in the database
#'
#' Checks a survey name against a list of all survey names in the DATRAS database.
#' If the name is not matched it puts up a message showing the available survey names.
#'
#' @param survey the survey acronym, e.g. NS-IBTS.
#'
#' @return logical.
#'
#' @seealso
#' \code{\link{checkSurveyYearOK}} and \code{\link{getSurveyYearQuarterList}}
#' also perform checks against the DATRAS database.
#'
#' \code{\link{icesDatras-package}} gives an overview of the package.
#'
#' @examples
#' checkSurveyOK(survey = "ROCKALL")
#' checkSurveyOK(survey = "NOTALL")
#'
#' @export

#' @importFrom utils capture.output
checkSurveyOK <- function(survey) {
# check survey name against available surveys
available_surveys <- getSurveyList()
Expand All @@ -13,9 +33,33 @@ checkSurveyOK <- function(survey) {
}
}


#' Check that a survey and year combination is in the database
#'
#' Checks a year and/or survey name against a list of all survey year combinations in the DATRAS database.
#' If the combination is not matched it puts up a message showing the available options.
#'
#' @param survey the survey acronym, e.g. NS-IBTS.
#' @param year the year of the survey, e.g. 2010.
#' @param checksurvey logical, should the survey name also be checked.
#'
#' @return logical.
#'
#' @seealso
#' \code{\link{checkSurveyOK}} and \code{\link{checkSurveyYearQuarterOK}}
#' also perform checks against the DATRAS database.
#'
#' \code{\link{icesDatras-package}} gives an overview of the package.
#'
#' @examples
#' checkSurveyYearOK(survey = "ROCKALL", 2015)
#' checkSurveyYearOK(survey = "ROCKALL", 2000)
#' checkSurveyYearOK(survey = "NOTALL", 2000)
#'
#' @export

checkSurveyYearOK <- function(survey, year, checksurvey = TRUE) {
# check year against available years for a given survey

if (checksurvey) {
# check survey name
if (!checkSurveyOK(survey)) return(FALSE)
Expand All @@ -31,16 +75,47 @@ checkSurveyYearOK <- function(survey, year, checksurvey = TRUE) {
}
}


#' Check that a survey, year and quarter combination is in the database
#'
#' Checks a quarter and/or year and/or survey name against a list of all survey year quarter
#' combinations in the DATRAS database.
#' If the combination is not matched it puts up a message showing the available options.
#'
#' @param survey the survey acronym, e.g. NS-IBTS.
#' @param year the year of the survey, e.g. 2010.
#' @param quarter the quarter of the year the survey took place, i.e. 1, 2, 3 or 4.
#' @param checksurvey logical, should the survey name also be checked.
#' @param checkyear logical, should the year also be checked.
#'
#' @return logical.
#'
#' @seealso
#' \code{\link{checkSurveyOK}} and \code{\link{checkSurveyYearOK}}
#' also perform checks against the DATRAS database.
#'
#' \code{\link{icesDatras-package}} gives an overview of the package.
#'
#' @examples
#' checkSurveyYearQuarterOK(survey = "ROCKALL", 2015, 3)
#' checkSurveyYearQuarterOK(survey = "ROCKALL", 2015, 1)
#' checkSurveyYearQuarterOK(survey = "ROCKALL", 2000, 1)
#' checkSurveyYearQuarterOK(survey = "NOTALL", 2000, 1)
#'
#' # be careful of unexpected results with checksurvey and checkyear!
#' checkSurveyYearQuarterOK(survey = "NOTALL", 2000, 1, checksurvey=FALSE)
#'
#' @export

checkSurveyYearQuarterOK <- function(survey, year, quarter, checksurvey = TRUE, checkyear = TRUE) {
# check year against available years for a given survey

if (checksurvey) {
# check survey name
if (!checkSurveyOK(survey)) return(FALSE)
}

if (checkyear) {
# check survey name
# check survey year
if (!checkSurveyYearOK(survey, year, checksurvey = FALSE)) return(FALSE)
}

Expand Down
31 changes: 11 additions & 20 deletions R/getCAdata.R
Original file line number Diff line number Diff line change
@@ -1,36 +1,30 @@
#' Age based data
#' Get Age-Based Data
#'
#' Returns age-based information such as sex, maturity, and age counts per length of sampled species.
#' Get age-based data such as sex, maturity, and age counts per length of sampled species.
#'
#' @param survey the survey accronym e.g. NS-IBTS, BITS.
#' @param year the numeric year of the survey, e.g. 2010.
#' @param survey the survey acronym, e.g. NS-IBTS.
#' @param year the year of the survey, e.g. 2010.
#' @param quarter the quarter of the year the survey took place, i.e. 1, 2, 3 or 4.
#'
#'
#' @return A data.frame.
#' @return A data frame.
#'
#' @seealso
#' \code{\link{getSurveyList}} returns the acronyms for available surveys.
#'
#' \code{\link{getSurveyYearList}} returns the years available for a given survey.
#' \code{\link{getDATRAS}} supports querying many years and quarters in one function call.
#'
#' \code{\link{getSurveyYearQuarterList}} returns the quarters available for a given survey and year.
#' \code{\link{getHHdata}} and \code{\link{getHLdata}} get haul data and
#' length-based data.
#'
#' \code{\link{icesDatras-package}} gives an overview of the package.
#'
#' @author Colin Millar.
#'
#' @examples
#' # read age data
#' cadata <- getCAdata(survey = "BTS", year = 2015, quarter = 3)
#' cadata <- getCAdata(survey = "ROCKALL", year = 2002, quarter = 3)
#' str(cadata)
#'
#'
#' @export

getCAdata <- function(survey, year, quarter) {
# Returns age-based information such as sex, maturity, and age counts per length of sampled species.

# check survey name
if (!checkSurveyOK(survey)) return(FALSE)

Expand All @@ -40,16 +34,13 @@ getCAdata <- function(survey, year, quarter) {
# check quarter
if (!checkSurveyYearQuarterOK(survey, year, quarter, checksurvey = FALSE, checkyear = FALSE)) return(FALSE)

# read and parse XML from api
# read url and parse to data frame
url <-
sprintf(
"https://datras.ices.dk/WebServices/DATRASWebService.asmx/getCAdata?survey=%s&year=%i&quarter=%i",
survey, year, quarter)
out <- curlDatras(url = url)
out <- readDatras(url)
out <- parseDatras(out)

# return
out
}


Loading