Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add support for SDMX #27

Closed
GregorDeCillia opened this issue Mar 23, 2022 · 2 comments
Closed

Add support for SDMX #27

GregorDeCillia opened this issue Mar 23, 2022 · 2 comments
Labels
parse
Milestone
Version 1.0

Comments

@GregorDeCillia
Copy link
Contributor

GregorDeCillia commented Mar 23, 2022 •
edited
Loading

It would be very useful if {STATcubeR} could support "SDMX archives" which are generated from STATcube. sdmx archives consist of a metadata component called the "structure definition" and a data part which contains the actual cell values. In order to support that, we would need to add parsers for the xml-based data format.

The generated archives are more or less compatible with the CRAN package rsdmx: https://cran.r-project.org/package=rsdmx, which could be used as a starting point to develop parsers.

Possible usuage: parser function sdmx_table() which generates an object of class sc_data (the parent class for OGD and STATcube-API datasets)

x <- STATcubeR::sdmx_table("path/to/sdmx_archive.zip")
class(x)
#> [2] "sdmx_table" "sc_data" "R6"

There are several advantages of the sdmx format compared to the API

  • The structure definition contains information about hierarchical classifications, which are not available via the API
  • The download option "sdmx archive" is available even if STATcube is used as a "guest user"
  • SDMX is used by other SuperSTAR products such as SuperCROSS

The last point is probably the most compelling one since a direct interface to SuperCROSS would be very helpful for the internal workflows of statistics austria
@GregorDeCillia GregorDeCillia added this to the Nice to have milestone Mar 23, 2022
@GregorDeCillia
Copy link
Contributor Author

Screenshot: SDMX downloads are available for unregistered users on the external STATcube instance

image

GregorDeCillia added a commit that referenced this issue Mar 29, 2023
@GregorDeCillia
add support for sdmx parsing
f4e18ac 
this is the first step to resolving
#27 by adding a function that creates
sc_table() like objects based on sdmx
archives

The sdmx format contains all metadata
that is necessary for STATcubeR to reuse
the existing $tabulate() workflow and this
first version already provides support for
various features via the base class (sc_data)

- $tabulate() to aggregate data
- $total_codes() to set/unset total codes
- $recoder to recode datasets (change labels)
  change codes, toggle visibility of
  elements, reorder elements, etc.
- importing german and english labels
  simultaniously (both languages are included
  in a zip download) and allowing to swhitch
  between them using $language<-().

New features
- sdmx arcives provide a $parent column
  in the $fields() table which are used
  to represent hierarchical classifications.
  Previously, this was only possible with
  od_table()

There are still some improvements. See
the issue #27 for more details

- properly parse time variables -
  currently they are treated as generic
  categories.
- parse element annotations (detailed
  descriptions for classification
  elements) and add them to
  $field()$de_desc just like with
  OGD dataset
- parse value annotations (see #39)
- provide a print/fromat method
- add a reasonable logic for total
  codes that takes the parent codes into
  account
- fill meta$measures$fun and
  $meta$measures$precision based on
  the sdmx metadata
- modify very long codes which use
  the @-symbol (probably for escapes)
- extend documentation
- possibly check SuperCROSS compability
@GregorDeCillia GregorDeCillia linked a pull request Apr 6, 2023 that will close this issue
Add {tibble} as a dependency #32
Closed
5 tasks
@GregorDeCillia GregorDeCillia pinned this issue Apr 6, 2023
@bernhard-da
Copy link
Collaborator

sdmx_table() is now exported

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
parse
Projects
None yet
Milestone
Version 1.0
Development

Successfully merging a pull request may close this issue.

Add {tibble} as a dependency statistikat/STATcubeR
2 participants
@bernhard-da @GregorDeCillia