issues #316 and #317

[ake123]

issues #316 and #317
created one function to do all the downloads from the links provided from data browser download:

• Tab separated values (.tsv)
• SDMX-CSV 1.0 (.csv)
• SDMX-CSV 2.0 (.csv)
• SDMX-ML 2.1 StructureSpecific (.xml)
• SDMX-ML 2.1 Generic (.xml)
• SDMX-ML 3.0 (.xml)
• JSON-stat (.json)

[pitkant]

Thanks, will test this shortly and provide feedback if needed.

[pitkant]

I tested this with the link https://ec.europa.eu/eurostat/api/dissemination/sdmx/2.1/data/ttr00011/A.T.FRM_LD_NLD.TOT.TOTAL.EU27_2020+EA20+EA19+BE+BG+CZ+DK+DE+EE+IE+EL+ES+FR+HR+IT+CY+LV+LT+LU+HU+MT+NL+AT+PL+PT+RO+SI+SK+FI+SE+IS+LI+NO+CH+UK+BA+ME+MK+AL+RS+TR?format=TSV&compressed=true&startPeriod=2013&endPeriod=2024

1. It would be good if the function used httr2 instead of httr
2. It would be great if the function utilized existing eurostat package functions. Now the output is pretty rough, as it doesn't parse the missing value : correctly and it does not interpret the first column of the tsv correctly and unpack the variables before year columns:

head(testi)
# A tibble: 6 × 13
  `freq,unit,tra_meas,schedule,tra_cov,geo\\TIME_PERIOD` `2013`      `2014`       `2015`       `2016`      `2017`      `2018`      `2019`      `2020`      `2021`      `2022`      `2023`      `2024`     
  <chr>                                                  <chr>       <chr>        <chr>        <chr>       <chr>       <chr>       <chr>       <chr>       <chr>       <chr>       <chr>       <chr>      
1 A,T,FRM_LD_NLD,TOT,TOTAL,AT                            "196459 "   "217785 "    "216621 "    "223421.2 " "227642.8 " "237700.6 " "228223.4 " "165013.2 " "194820.2 " "188182.2 " "190293.5 " "227789.7 "
2 A,T,FRM_LD_NLD,TOT,TOTAL,BA                            ": "        ": "         ": "         ": "        ": "        ": "        ": "        ": "        "11518.7 "  "16195 "    "17709 "    ": "       
3 A,T,FRM_LD_NLD,TOT,TOTAL,BE                            "957012.2 " "1014585.6 " "1126234.1 " "1088734 "  "1251173 "  "1416428 "  "1397513 "  "1584640 "  "2083632 "  "1729332 "  "1519106 "  "1714273 " 
4 A,T,FRM_LD_NLD,TOT,TOTAL,BG                            "19574.9 "  "23072.8 "   "31520 "     "32976.9 "  "34806.7 "  "29867.3 "  "28748.6 "  "25844 "    "25686 "    "26457 "    "21957.3 "  "22481.2 " 
5 A,T,FRM_LD_NLD,TOT,TOTAL,CH                            "400028.4 " "407703.6 "  "401924.6 "  "428495.5 " "486164.2 " "493201.9 " "459877 "   "272379.7 " "341402.7 " "364304.6 " "359146.6 " "411705.9 "
6 A,T,FRM_LD_NLD,TOT,TOTAL,CY                            "28327.9 "  "28095 "     "27626.2 "   "28430 "    "30880.4 "  "32186.1 "  "32360.1 "  "24006.6 "  "25149.5 "  "25141.9 "  "26702.4 "  "31851.4 " 

[pitkant]

The correct data query parameter is compressed=true, not compress=true, see docs: https://ec.europa.eu/eurostat/web/user-guides/data-browser/api-data-access/api-getting-started/sdmx2.1#APIGettingstartedwithSDMX2.1API-Compression

[pitkant]

The issue with "compress=true" on row 28 is still not resolved with commit ea5d4c9.

[ake123]

Yes you are correct but I tested by copying the API link from the downloads button here. https://ec.europa.eu/eurostat/databrowser/view/ttr00011/default/table?lang=en&category=t_avia

the TSV link is this
https://ec.europa.eu/eurostat/api/dissemination/sdmx/3.0/data/dataflow/ESTAT/ttr00011/1.0/*.*.*.*.*.*?c[freq]=A&c[unit]=T&c[tra_meas]=FRM_LD_NLD&c[schedule]=TOT&c[tra_cov]=TOTAL&c[geo]=EU27_2020,BE,BG,CZ,DK,DE,EE,IE,EL,ES,FR,HR,IT,CY,LV,LT,LU,HU,MT,NL,AT,PL,PT,RO,SI,SK,FI,SE,IS,NO,CH,UK,BA,ME,MK,RS,TR,AL,EA19,EA20,LI&c[TIME_PERIOD]=2013,2014,2015,2016,2017,2018,2019,2020,2021,2022,2023,2024&compress=true&format=tsv

with compress=true which is a bit different from the documentation.

[pitkant]

Ah, yes, you're right that the SDMX 2.1 and SDMX 3.0 API logic seems to be different. 3.0 will probably be the future but I think it makes sense to support 2.1 as well.

Ok, I looked at the API urls provided by the data browser and indeed it seems that all download links - even the SDMX-ML 2.1 StructureSpecific (.xml) and SDMX-ML 2.1 Generic (.xml) point to 3.0 endpoint and then add &formatVersion=2.1 at the end. Weird but if it works it works. Seems like prioritising support for 3.0 conventions is the way to go, the syntax to adding filters seems to also be more flexible than with 2.1.