Skip to content

Latest commit

 

History

History
executable file
·
2225 lines (2149 loc) · 56.7 KB

README.md

File metadata and controls

executable file
·
2225 lines (2149 loc) · 56.7 KB
Raw

modisfast

licence CRAN_Status_Badge Github_Status_Badge R-CMD-check DOI CRAN_downloads

Table of contents

• Overview
• Installation
• Get started
• Data collections available
• Manual testing of the functionality
• Foundational framework
• Comparison with similar R packages
• Citation
• Future developments
• Contributing
• Acknowledgments

Overview

modisfast is an R package designed for easy and fast downloads of MODIS Land products, VIIRS Land products, and GPM (Global Precipitation Measurement Mission) Earth Observation data.

modisfast uses the abilities offered by the OPeNDAP framework (Open-source Project for a Network Data Access Protocol) to download a subset of Earth Observation data cube, along spatial, temporal or any other data dimension (depth, …). This way, it reduces downloading time and disk usage to their minimum : no more 1° x 1° MODIS tiles with 10 bands when your region of interest is only 30 km x 30 km wide and you need 2 bands ! Moreover, modisfast enables parallel downloads of data.

This package is hence particularly suited for retrieving MODIS or VIIRS data over long time series and over areas, rather than short time series and points.

Importantly, the robust, sustainable, and cost-free foundational framework of modisfast, both for the data provider (NASA) and the software (R, OPeNDAP, the tidyverse and GDAL suite of packages and software), guarantees the long-term reliability and open-source nature of the package.

By enabling to download subsets of data cubes, modisfast facilites the access to Earth science data for R users in places where internet connection is slow or expensive and promotes digital sobriety for our research work.

Installation

You can install the released version of modisfast from CRAN with :

install.packages("modisfast")

or the development version (to get a bug fix or to use a feature from the development version) with :

if(!require(devtools)){install.packages("devtools")}
devtools::install_github("ptaconet/modisfast")

Get Started

Accessing and opening MODIS data with modisfast is a simple 3-steps workflow. This example shows how to download and import a one-year-long monthly time series of MODIS Normalized Difference Vegetation Index (NDVI) at 1 km spatial resolution over the whole country of Madagascar.

1/ First, define the variables of interest (ROI, time frame, collection, and bands) :

# Load the packages
library(modisfast)
library(sf)
library(terra)

# ROI and time range of interest
roi <- st_as_sf(data.frame(id = "madagascar", geom = "POLYGON((41.95 -11.37,51.26 -11.37,51.26 -26.17,41.95 -26.17,41.95 -11.37))"), wkt = "geom", crs = 4326) # a ROI of interest, format sf polygon
time_range <- as.Date(c("2023-01-01", "2023-12-31")) # a time range of interest

# MODIS collections and variables (bands) of interest
collection <- "MOD13A3.061" # run mf_list_collections() for an exhaustive list of collections available
variables <- c("_1_km_monthly_NDVI") # run mf_list_variables("MOD13A3.061") for an exhaustive list of variables available for the collection "MOD13A3.061"

2/ Then, get the URL of the data and download them :

## Login to Earthdata servers with your EOSDIS credentials.
# To create an account (free) go to : https://urs.earthdata.nasa.gov/.
log <- mf_login(credentials = c("username", "password")) # set your own EOSDIS username and password

## Get the URLs of the data
urls <- mf_get_url(
  collection = collection,
  variables = variables,
  roi = roi,
  time_range = time_range
)

## Download the data. By default the data is downloaded in a temporary directory, but you can specify a folder
res_dl <- mf_download_data(urls, parallel = TRUE)

3/ And finally, import the data in R as a terra::SpatRaster object using the function mf_import_data() ( ⚠️ see here why you should use this function, instead of the original terra::rast(), in the context of modisfast) :

r <- mf_import_data(
  path = dirname(res_dl$destfile[1]),
  collection = collection,
  proj_epsg = 4326
)

terra::plot(r, col = rev(terrain.colors(20)))

Time series of monthly 1-km MODIS NDVI over Madagascar for the year 2023, retrieved with modisfast

Time series of monthly 1-km MODIS NDVI over Madagascar for the year 2023, retrieved with modisfast

et voilà !

Want more examples ? modisfast provides three long-form documentations and examples to learn more about the package :

Collections available in modisfast

Currently modisfast supports download of 77 data collections, extracted from the following meta-collections :

Details of each product available for download are provided in the tables below or through the function mf_list_collections().

MODIS and VIIRS data collections accessible with modisfast (click to expand)

Collection Source Type Name Spatial resolution Temporal resolution Temporal extent
MCD43A1.061 MODIS Albedo MODIS/Terra and Aqua BRDF/Albedo Model Parameters Daily L3 Global 500 m SIN Grid 500 m Daily 2000-02-24 to present
MCD43A2.061 MODIS Albedo MODIS/Terra and Aqua BRDF/Albedo Quality Daily L3 Global 500 m SIN Grid 500 m Daily 2000-02-24 to present
MCD43A3.061 MODIS Albedo MODIS/Terra and Aqua Albedo Daily L3 Global 500 m SIN Grid 500 m Daily 2000-02-24 to present
VNP43MA2.001 VIIRS Albedo VIIRS/NPP BRDF/Albedo Quality Daily L3 Global 1 km SIN Grid 1000 m Daily 2012-01-17 to present
VNP43MA3.001 VIIRS Albedo VIIRS/NPP Albedo Daily L3 Global 1 km SIN Grid 1000 m Daily 2012-01-17 to present
VNP43MA1.001 VIIRS Albedo VIIRS/NPP BRDF/Albedo Model Parameters Daily L3 Global 1 km SIN 1000 m Daily 2012-01-17 to present
VNP43IA2.001 VIIRS Albedo VIIRS/NPP BRDF/Albedo Quality Daily L3 Global 500 m SIN Grid 500 m Daily 2012-01-17 to present
VNP43IA3.001 VIIRS Albedo VIIRS/NPP Albedo Daily L3 Global 500 m SIN Grid 500 m Daily 2012-01-17 to present
MCD64A1.061 MODIS Burned areas MODIS/Terra+Aqua Burned Area Monthly L3 Global 500 m SIN Grid 500 m 30 day 2000-11-01 to present
MOD16A2GF.061 MODIS Evapotranspiration MODIS/Terra Net Evapotranspiration Gap-Filled 8-Day L4 Global 500 m SIN Grid 500 m 8 day 2000-01-01 to present
MYD16A2GF.061 MODIS Evapotranspiration MODIS/Aqua Net Evapotranspiration Gap-Filled 8-Day L4 Global 500 m SIN Grid 500 m 8 day 2002-01-01 to present
MOD16A2.061 MODIS Evapotranspiration MODIS/Terra Net Evapotranspiration 8-Day L4 Global 500m SIN Grid v061 500 m 8 day 2001-01-01 to present
MOD16A3GF.061 MODIS Evapotranspiration MODIS/Terra Net Evapotranspiration Gap-Filled Yearly L4 Global 500 m SIN Grid 500 m 365 day 2000-02-18 to present
MYD16A2.061 MODIS Evapotranspiration MODIS/Aqua Net Evapotranspiration 8-Day L4 Global 500m SIN Grid v061 500 m 8 day 2002-07-04 to present
MYD16A3GF.061 MODIS Evapotranspiration MODIS/Aqua Net Evapotranspiration Gap-Filled Yearly L4 Global 500 m SIN Grid 500 m 365 day 2002-07-04 to present
MCD12Q1.061 MODIS Land cover MODIS/Terra+Aqua Land Cover Type Yearly L3 Global 500 m SIN Grid 500 m 365 day 2001-01-01 to present
MOD11A2.061 MODIS Land surface temperature MODIS/Terra Land Surface Temperature/Emissivity 8-Day L3 Global 1 km SIN Grid v061 1000 m 8 day 2000-02-18 to present
MOD11A1.061 MODIS Land surface temperature MODIS/Terra Land Surface Temperature/Emissivity Daily L3 Global 1km SIN Grid v061 1000 m Daily 2000-02-24 to present
MYD11A2.061 MODIS Land surface temperature MODIS/Aqua Land Surface Temperature/Emissivity 8-Day L3 Global 1 km SIN Grid v061 1000 m 8 day 2002-07-04 to present
MYD11A1.061 MODIS Land surface temperature MODIS/Aqua Land Surface Temperature/Emissivity Daily L3 Global 1km SIN Grid v061 1000 m Daily 2002-07-05 to present
MOD11B2.061 MODIS Land surface temperature MODIS/Terra Land Surface Temperature/Emissivity 8-Day L3 Global 6 km SIN Grid 6000 m 8 day 2000-02-18 to present
MOD11B3.061 MODIS Land surface temperature MODIS/Terra Land Surface Temperature/Emissivity Monthly L3 Global 6 km SIN Grid 6000 m 30 day 2000-02-01 to present
MYD11B2.061 MODIS Land surface temperature MODIS/Aqua Land Surface Temperature/Emissivity 8-Day L3 Global 6 km SIN Grid 6000 m 8 day 2002-07-04 to present
MYD11B3.061 MODIS Land surface temperature MODIS/Aqua Land Surface Temperature/Emissivity Monthly L3 Global 6 km SIN Grid 6000 m 30 day 2002-07-01 to present
VNP21A2.001 VIIRS Land surface temperature VIIRS/NPP Land Surface Temperature and Emissivity 8-Day L3 Global 1 km SIN Grid 1000 m 8 day 2012-01-25 to present
VNP21A1N.001 VIIRS Land surface temperature VIIRS/NPP Land Surface Temperature and Emissivity Daily L3 Global 1 km SIN Grid Night 1000 m Daily 2012-01-19 to present
VNP21A1D.001 VIIRS Land surface temperature VIIRS/NPP Land Surface Temperature and Emissivity Daily L3 Global 1 km SIN Grid Day 1000 m Daily 2012-01-19 to present
MOD15A2H.061 MODIS Leaf area index MODIS/Terra Leaf Area Index/FPAR 8-Day L4 Global 500 m SIN Grid 500 m 8 day 2000-02-18 to present
MYD15A2H.061 MODIS Leaf area index MODIS/Aqua Leaf Area Index/FPAR 8-Day L4 Global 500 m SIN Grid 500 m 8 day 2002-07-04 to present
MCD15A2H.061 MODIS Leaf area index MODIS/Terra+Aqua Leaf Area Index/FPAR 8-Day L4 Global 500 m SIN Grid 500 m 8 day 2002-07-04 to present
MCD15A3H.061 MODIS Leaf area index MODIS/Terra+Aqua Leaf Area Index/FPAR 4-Day L4 Global 500 m SIN Grid 500 m 4 day 2002-07-04 to present
VNP15A2H.001 VIIRS Leaf area index VIIRS/NPP Leaf Area Index/FPAR 8-Day L4 Global 500 m SIN Grid 500 m 8 day 2012-01-17 to present
MODOCGA.061 MODIS Ocean Reflectance MODIS/Terra Ocean Reflectance Daily L2G-Lite Global 1 km SIN Grid 1000 m Daily 2000-02-24 to present
MYDOCGA.061 MODIS Ocean Reflectance MODIS/Aqua Ocean Reflectance Daily L2G-Lite Global 1 km SIN Grid 1000 m Daily 2002-07-04 to present
MOD17A3HGF.061 MODIS Primary Productivity MODIS/Terra Net Primary Production Gap-Filled Yearly L4 Global 500 m SIN Grid 500 m 365 day 2000-02-18 to present
MYD17A3HGF.061 MODIS Primary Productivity MODIS/Aqua Net Primary Production Gap-Filled Yearly L4 Global 500 m SIN Grid 500 m 365 day 2002-07-04 to present
MOD17A2H.061 MODIS Primary Productivity MODIS/Aqua Gross Primary Productivity 8-Day L4 Global 500 m SIN Grid 500 m 8 day 2000-02-18 to present
MOD17A2HGF.061 MODIS Primary Productivity MODIS/Terra Gross Primary Productivity Gap-Filled 8-Day L4 Global 500 m SIN Grid 500 m 8 day 2000-01-01 to present
MYD17A2H.061 MODIS Primary Productivity MODIS/Terra Gross Primary Productivity 8-Day L4 Global 500 m SIN Grid 500 m 8 day 2002-07-04 to present
MYD17A2HGF.061 MODIS Primary Productivity MODIS/Aqua Gross Primary Productivity Gap-Filled 8-Day L4 Global 500 m SIN Grid 500 m 8 day 2002-01-01 to present
MYD09GA.061 MODIS Surface reflectance MODIS/Aqua Surface Reflectance Daily L2G Global 1 km and 500 m SIN Grid 1000 m Daily 2002-07-04 to present
MOD09GA.061 MODIS Surface reflectance MODIS/Terra Surface Reflectance Daily L2G Global 1 km and 500 m SIN Grid 1000 m Daily 2000-02-24 to present
MOD09GQ.061 MODIS Surface reflectance MODIS/Terra Surface Reflectance Daily L2G Global 250 m SIN Grid 250 m Daily 2000-02-24 to present
MYD09GQ.061 MODIS Surface reflectance MODIS/Aqua Surface Reflectance Daily L2G Global 250 m SIN Grid 250 m Daily 2002-07-04 to present
MYD09Q1.061 MODIS Surface reflectance MODIS/Aqua Surface Reflectance 8-Day L3 Global 250 m SIN Grid 250 m 8 day 2002-07-04 to present
MOD09Q1.061 MODIS Surface reflectance MODIS/Terra Surface Reflectance 8-Day L3 Global 250 m SIN Grid 250 m 8 day 2000-02-18 to present
MYD09A1.061 MODIS Surface reflectance MODIS/Aqua Surface Reflectance 8-Day L3 Global 500 m SIN Grid 500 m 8 day 2002-07-04 to present
MCD43A4.061 MODIS Surface reflectance MODIS/Terra and Aqua Nadir BRDF-Adjusted Reflectance Daily L3 Global 500 m SIN Grid 500 m Daily 2000-02-24 to present
MOD09A1.061 MODIS Surface reflectance MODIS/Terra Surface Reflectance 8-Day L3 Global 500 m SIN Grid 500 m 8 day 2000-02-18 to present
VNP09A1.001 VIIRS Surface reflectance VIIRS/NPP Surface Reflectance 8-Day L3 Global 1 km SIN Grid 1000 m 8 day 2012-01-17 to present
VNP43MA4.001 VIIRS Surface reflectance VIIRS/NPP Nadir BRDF-Adjusted Reflectance Daily L3 Global 1 km SIN 1000 m Daily 2012-01-17 to present
VNP09H1.001 VIIRS Surface reflectance VIIRS/NPP Surface Reflectance 8-Day L3 Global 500 m SIN Grid 500 m 8 day 2012-01-17 to present
VNP43IA4.001 VIIRS Surface reflectance VIIRS/NPP Nadir BRDF-Adjusted Reflectance Daily L3 Global 500 m SIN Grid 500 m Daily 2012-01-17 to present
VNP14A1.001 VIIRS Thermal Anomalies/Fire VIIRS/NPP Thermal Anomalies/Fire Daily L3 Global 1 km SIN Grid 1000 m Daily 2012-01-19 to present
MODTBGA.061 MODIS Thermal Bands MODIS/Terra Thermal Bands Daily L2G-Lite Global 1 km SIN Grid 1000 m Daily 2000-02-24 to present
MYDTBGA.061 MODIS Thermal Bands MODIS/Aqua Thermal Bands Daily L2G-Lite Global 1 km SIN Grid 1000 m Daily 2002-07-04 to present
MOD13A2.061 MODIS Vegetation indices MODIS/Terra Vegetation Indices 16-Day L3 Global 1 km SIN Grid 1000 m 16 day 2000-02-18 to present
MOD13A3.061 MODIS Vegetation indices MODIS/Terra Vegetation Indices Monthly L3 Global 1 km SIN Grid 1000 m 30 day 2000-02-01 to present
MYD13A2.061 MODIS Vegetation indices MODIS/Aqua Vegetation Indices 16-Day L3 Global 1 km SIN Grid 1000 m 16 day 2002-07-04 to present
MYD13A3.061 MODIS Vegetation indices MODIS/Aqua Vegetation Indices Monthly L3 Global 1 km SIN Grid 1000 m 30 day 2002-07-01 to present
MOD13Q1.061 MODIS Vegetation indices MODIS/Terra Vegetation Indices 16-Day L3 Global 250m SIN Grid v061 250 m 16 day 2000-02-18 to present
MYD13Q1.061 MODIS Vegetation indices MODIS/Aqua Vegetation Indices 16-Day L3 Global 250m SIN Grid v061 250 m 16 day 2002-07-04 to present
MOD13A1.061 MODIS Vegetation indices MODIS/Terra Vegetation Indices 16-Day L3 Global 500 m SIN Grid 500 m 16 day 2000-02-18 to present
MYD13A1.061 MODIS Vegetation indices MODIS/Aqua Vegetation Indices 16-Day L3 Global 500 m SIN Grid 500 m 16 day 2002-07-04 to present
VNP13A2.001 VIIRS Vegetation indices VIIRS/NPP Vegetation Indices 16-Day L3 Global 1 km SIN Grid 1000 m 16 day 2012-01-17 to present
VNP13A3.001 VIIRS Vegetation indices VIIRS/NPP Vegetation Indices Monthly L3 Global 1 km SIN Grid 1000 m 30 day 2012-01-01 to present
VNP13A1.001 VIIRS Vegetation indices VIIRS/NPP Vegetation Indices 16-Day L3 Global 500 m SIN Grid 500 m 16 day 2012-01-17 to present

Other (non-MODIS or VIIRS) data collections accessible with modisfast (click to expand)

Collection Source Type Name Spatial resolution Temporal resolution Temporal extent
GPM_3IMERGDE.06 GPM Rainfall GPM IMERG Early Precipitation L3 1 day 0.1 degree x 0.1 degree V06 10000 m Daily 2000-06-01 to present
GPM_3IMERGDF.06 GPM Rainfall GPM IMERG Final Precipitation L3 1 day 0.1 degree x 0.1 degree V06 10000 m Daily 2000-06-01 to present
GPM_3IMERGDF.07 GPM Rainfall GPM IMERG Final Precipitation L3 1 day 0.1 degree x 0.1 degree V07 10000 m Daily 2000-06-01 to present
GPM_3IMERGDL.06 GPM Rainfall GPM IMERG Late Precipitation L3 1 day 0.1 degree x 0.1 degree V06 10000 m Daily 2000-06-01 to present
GPM_3IMERGHH.06 GPM Rainfall GPM IMERG Final Precipitation L3 Half Hourly 0.1 degree x 0.1 degree V06 10000 m 30 minute 2000-06-01 to present
GPM_3IMERGHH.07 GPM Rainfall GPM IMERG Final Precipitation L3 Half Hourly 0.1 degree x 0.1 degree V07 10000 m 30 minute 2000-06-01 to present
GPM_3IMERGHHE.06 GPM Rainfall GPM IMERG Early Precipitation L3 Half Hourly 0.1 degree x 0.1 degree V06 10000 m 30 minute 2000-06-01 to present
GPM_3IMERGHHL.06 GPM Rainfall GPM IMERG Late Precipitation L3 Half Hourly 0.1 degree x 0.1 degree V06 10000 m 30 minute 2000-06-01 to present
GPM_3IMERGM.06 GPM Rainfall GPM IMERG Final Precipitation L3 1 month 0.1 degree x 0.1 degree V06 10000 m 1 month 2000-06-01 to present
GPM_3IMERGM.07 GPM Rainfall GPM IMERG Final Precipitation L3 1 month 0.1 degree x 0.1 degree V07 10000 m 1 month 2000-06-01 to present

Manual testing of the functionality

Since most modisfast functions depend on EarthData credentials, automated tests are disabled. However, after installation, users can manually test the package’s functionality by running these lines of code :

# replace "username" and "password" with your own EOSDIS (Earthdata) credentials 
earthdata_un <- "username" 
earthdata_pw <- "password"

devtools::test("modisfast")

Foundational framework

Technically, modisfast is a programmatic interface (R wrapper) to several NASA OPeNDAP servers. OPeNDAP is the acronym for Open-source Project for a Network Data Access Protocol and designates both the software, the access protocol, and the corporation that develops them. The OPeNDAP is designed to simplify access to structured and high-volume data, such as satellite products, over the Web. It is a collaborative effort involving multiple institutions and companies, with open-source code, free software, and adherence to the Open Geospatial Consortium (OGC) standards. It is widely used by NASA, which partly finances it.

A key feature of OPeNDAP is its capability to apply filters at the data download process, ensuring that only the necessary data is retrieved. These filters, specified within a URL, can be spatial, temporal, or dimensional. Although powerful, OPeNDAP URLs are not trivial to build. modisfast facilitates this process by constructing the URL based on the spatial, temporal, and dimensional filters provided by the user in the function mf_get_url().

Let’s take an example to understand.

The following URL ⬇️

https://opendap.cr.usgs.gov/opendap/hyrax/MOD11A2.061/h17v08.ncml.nc4?MODIS_Grid_8Day_1km_LST_eos_cf_projection,LST_Day_1km[775:793][55:140][512:560],LST_Night_1km[775:793][55:140][512:560],QC_Day[775:793][55:140][512:560],QC_Night[775:793][55:140][512:560],time[775:793],YDim[55:140],XDim[512:560]

is a link to download the following subset of MOD11A2.061 data in netCDF :

  • bands LST_Day_1km, LST_Night_1km, QC_Day, QC_Night ;
  • each available date between the 2017-01-01 and the 2017-06-01 ;
  • within the following bounding box (lon/lat): -5.41 8.84, -5.82 9.54.

The indices within the [] refer to values encoding for the spatial and temporal filters.

These OPeNDAP URLs are not trivial to build. modisfast converts the spatial, temporal and dimensional filters (R objects) provided by the user through the function mf_get_url() into the appropriate OPeNDAP URL(s). Subsequently, the function mf_download_data() allows for downloading the data using the httr and parallel packages.

Comparison with similar R packages

There are other R packages available for accessing MODIS data. Below is a comparison of modisfast with other packages available for downloading chunks of MODIS or VIIRS data :

Package Data Available on CRAN Utilizes open standards for data access protocols Spatial subsetting* Dimensional subsetting* Maximum area size allowed for download Speed**
modisfast MODIS, VIIRS, GPM ✅ ✅ ✅ ✅ unlimited ✅
appeears MODIS, VIIRS, and many others ✅ ✅ ✅ ✅ unlimited variable
MODISTools MODIS, VIIRS ✅ ❌ ✅ ✅ 200 km x 200 km ✅
rgee MODIS, VIIRS, GPM, and many others ✅ ❌ ✅ ✅ unlimited not tested
MODIStsp MODIS ❌ ❌ ✅ unlimited NA
MODIS MODIS ❌ ❌ ❌ ❌ NA NA

* at the downloading phase

** Take a look at the article “Comparison of performance with other similar R packages” to get an overview of how modisfast compares to these packages in terms of data access time.

Citation

This package is licensed under a GNU General Public License v3.0 or later license.

We thank in advance people that use modisfast for citing it in their work / publication(s). For this, please use the following citation :

Taconet et al., (2024). modisfast: An R package for fast and efficient access to MODIS, VIIRS and GPM Earth Observation data. Journal of Open Source Software, 9(103), 7343, https://doi.org/10.21105/joss.07343

Future developments

Future developments of the package may include access to additional data collections from other OPeNDAP servers, and support for a variety of data formats as they become available from data providers through their OPeNDAP servers. Furthermore, the creation of an RShiny application on top of the package is being considered, as a means of further simplifying data access for users with limited coding skills.

Contributing

All types of contributions are encouraged and valued. For more information, check out our Contributor Guidelines.

Please note that the modisfast project is released with a Contributor Code of Conduct. By contributing to this project, you agree to abide by its terms.

Acknowledgments

We thank NASA and its partners for making all their Earth science data freely available, and implementing open data access protocols such as OPeNDAP. modisfast heavily builds on top of the OPeNDAP, so we thank the non-profit OPeNDAP, Inc. for developing the eponym tool in an open and collaborative way.

We also thank the contributors that have tested the package, reviewed the documentation and brought valuable feedbacks to improve the package : Florian de Boissieu, Julien Taconet.

This work has been developed over the course of several research projects (REACT 1, REACT 2, ANORHYTHM and DIV-YOO) funded by Expertise France, the French National Research Agency (ANR), and the French National Research Institute for Sustainable Development (IRD).