From 64652404553661f32034bb91d17201ddff99853b Mon Sep 17 00:00:00 2001 From: klau506 Date: Mon, 7 Oct 2024 16:40:48 +0200 Subject: [PATCH] update vignettes --- .../Modify_Mapping_Template_Tutorial.Rmd | 98 ++++--------- vignettes/Reporting_Variables.Rmd | 130 +++++++++--------- 2 files changed, 89 insertions(+), 139 deletions(-) diff --git a/vignettes/Modify_Mapping_Template_Tutorial.Rmd b/vignettes/Modify_Mapping_Template_Tutorial.Rmd index 8864a31f..b96f2f0e 100644 --- a/vignettes/Modify_Mapping_Template_Tutorial.Rmd +++ b/vignettes/Modify_Mapping_Template_Tutorial.Rmd @@ -44,66 +44,16 @@ Loading data, performing checks, and saving output... [1] "ag_demand_clean" Error in left_join_strict(., filter_variables(get(paste("ag_demand_map", : - Error: Some rows in the left dataset do not have matching keys in the right dataset. + Error: Some rows in the left dataset do not have matching keys in the right dataset. In particular, the mapping ag_demand_map_v7.0 misses the following rows: + inputs, sector + newproduct1, FoodDemand_Staples + newproduct2, FoodDemand_Staples + newproduct3, FoodDemand_NonStaples ``` -This error indicates that the mapping file `ag_demand_map` does not contain all the required items when computing the `ag_demand_clean` variable. To resolve this, follow these steps: +This error indicates that the mapping file `ag_demand_map` does not contain all the required items when computing the `ag_demand_clean` variable. To resolve this error, add the missing items (indicated in the terminal) to the mapping file, in this example located at `inst\extdata\mappings\GCAM7.0\ag_demand_map.csv`. Save the changes and close the file. -   3.1) Locate the variable generation code: - -     a) Open the file `R/functions.R` in the `gcamreport` folder. - -     b) Search (`Ctrl + F`) for the variable `ag_demand_clean`, which is generated within the `get_ag_demand()` function. - -   3.2) Identify the missing items: - -     a) Navigate to the section of the code that uses the `left_join_strict()` function: - - ```r - dplyr::bind_rows( - rgcam::getQuery(prj, "demand balances by crop commodity"), - rgcam::getQuery(prj, "demand balances by meat and dairy commodity") - ) %>% - # Adjust OtherMeat_Fish - dplyr::mutate(sector = dplyr::if_else(sector == "FoodDemand_NonStaples" & input == "OtherMeat_Fish", "OtherMeat_Fish", sector)) %>% - left_join_strict(filter_variables(get(paste('ag_demand_map',GCAM_version,sep='_'), envir = asNamespace("gcamreport")), "ag_demand_clean"), - by = c("sector"), multiple = "all") - ... - ``` - -     b) Add a break point at the beginning of the section and re-run the `generate_report` function again, this time pointing to your project file instead of the database to avoid regenerating the project and to stop at the breaking point: - -```r -generate_report(prj_name = '../path/to/your/dbname_of_the_project.dat', - scenarios = c('scenarios','list'), - final_year = XXX, GCAM_version = 'v7.0') -``` - -     c) To find the missing items, assign the lines before the `left_join_strict` function to a variable and the mapping items to another variable. Then, compare the two by `sector` as intended by the `left_join_strict` function. Use the following example: - - ```r - GCAM_version <- "v7.0" - - left_df <- dplyr::bind_rows( - rgcam::getQuery(prj, "demand balances by crop commodity"), - rgcam::getQuery(prj, "demand balances by meat and dairy commodity") - ) %>% - # Adjust OtherMeat_Fish - dplyr::mutate(sector = dplyr::if_else(sector == "FoodDemand_NonStaples" & input == "OtherMeat_Fish", "OtherMeat_Fish", sector)) - - right_df <- filter_variables(get(paste('ag_demand_map',GCAM_version,sep='_'), envir = asNamespace("gcamreport")), "ag_demand_clean") - - result <- dplyr::left_join(left_df, right_df, by = c("sector"), multiple = "all") - - unmatched <- result %>% - dplyr::filter(if_any(-one_of(names(left_df)), is.na)) - - View(unmatched) - ``` - -   3.3) Update the pre-build mappings: - -     a) Add the missing items to the `ag_demand_map` mapping file. Note that mapping files can sometimes be saved under different names. To find the correct file, open `inst/extdata/saveDataFiles_GCAM7.0.R` and search (`Ctrl + F`) for `ag_demand_map`: +Note that mapping files can sometimes be saved under different names. To find the correct file name, open `inst/extdata/saveDataFiles_GCAM7.0.R` and search (`Ctrl + F`) for `ag_demand_map` to find the full path. In this example: ```r ag_demand_map_v7.0 <- read.csv(file.path(rawDataFolder, "inst/extdata/mappings/GCAM7.0", "ag_demand_map.csv"), @@ -111,23 +61,16 @@ generate_report(prj_name = '../path/to/your/dbname_of_the_project.dat', ``` - In this example, the mapping file is located at `inst/extdata/mappings/GCAM7.0/ag_demand_map.csv`. Open the file, add the missing items, and save the changes. +If the error message is unclear and does not provide the mapping file name, consider debugging by setting breakpoints to the chunk where the variable is created. In this example, to the `ag_demand_clean` variable chunk. - - **Note**: If you don't want to include certain items in the report, you can add a line with the new item indicating `NoReported` as variable: + **Note**: If you don't want to include certain items in the report, you can add a line with the new item indicating `NoReported` as variable. In our example: ```r - newItemName,NoReported,,,,,,,,1 + newproduct1,FoodDemand_Staples,NoReported,,,,,,,,1 ``` - - **Note**: If the error persists and you prefer not to update the mapping file further, you can modify the `left_join_strict` call in the code to `dplyr::left_join.` However, **please be aware that this approach is not recommended, as it may compromise the strict matching intended in the data processing**. If you choose this option, please follow the next Step 3.4, source the `R/functions.R` file, and use Step 3.5 to ensure the new chuck will be read. - -   3.4) Stop the debugging process. - - -   3.5) Update the package data: +4) Update the package data:      a) Run the `inst/extdata/saveDataFiles_GCAM7.0.R` script to update the package data. @@ -135,7 +78,7 @@ generate_report(prj_name = '../path/to/your/dbname_of_the_project.dat',      c) Install the updated package by going to `Build > Install`. -4) Run the `generate_report` function again, this time pointing to your project file instead of the database to avoid regenerating the project: +5) Run the `generate_report` function again, this time pointing to your project file instead of the database to avoid regenerating the project: ```r generate_report(prj_name = '../path/to/your/dbname_of_the_project.dat', @@ -143,7 +86,7 @@ generate_report(prj_name = '../path/to/your/dbname_of_the_project.dat', final_year = XXX, GCAM_version = 'v7.0') ``` -The reporting procedure will start immediately and if the `ag_demand_map` mapping was arranged correctly, a new error will prompt regarding the `ag_prices_map`: +The reporting procedure will start immediately and if the `ag_demand_map` mapping was arranged correctly, a new error might prompt regarding the `ag_prices_map`: ```r Loading data, performing checks, and saving output... @@ -151,15 +94,21 @@ Loading data, performing checks, and saving output... [1] "ag_prices_clean" Error in left_join_strict(., filter_variables(get(paste("ag_prices_map", : - Error: Some rows in the left dataset do not have matching keys in the right dataset. + Error: Some rows in the left dataset do not have matching keys in the right dataset. In particular, the mapping ag_prices_map_v7.0 misses the following rows: + sector + newproduct1 + newproduct2 + newproduct3 ``` -Repeat the procedure outlined in Step 3 to correct the `ag_prices_map` file. +Add the missing items to the mapping located at `inst\extdata\mappings\GCAM7.0\ag_prices_map.csv`. Save the changes to the file and repeat the procedure detailed in Step 4. -5) Iterate Step 4 until the standardization process is complete. Occasionally, it may be necessary to clear all environment variables and restart R to ensure the updated mapping files are properly loaded. +6) Iterate Step 4 and 5 until the standardization process is complete. Occasionally, it may be necessary to clear all environment variables and restart R to ensure the updated mapping files are properly loaded. +**Note**: If the error persists and you prefer not to update the mapping files further, you can change the `left_join_strict` call in the code to `dplyr::left_join.` However, **please be aware that this approach is not recommended, as it may compromise the strict matching intended in the data processing**. If you choose this option, please follow Step 4 sourcing the `R/functions.R` file to ensure the new chuck will be read. -6) Do not forget to push the changes to your branch and tag the new version to allow reproducibility and reusability!! :) + +7) Do not forget to push the changes to your branch and tag the new version to allow reproducibility and reusability!! :) **Note**: If you want to install this `gcamreport` version into other devices, indicate the tag or branch name when cloning the repository, for instance: @@ -192,6 +141,7 @@ devtools::install_github('bc3LC/gcamreport@vUpdated') + ## Example 2: step-by-step to add a new mapping item - *AvocadoSeeds* and *AvocadoSeedsTrees* In this example, we assume that the GCAM version is a modification of the GCAM core 7.0 version with a more detailed land-food system which reports a new food item called *AvocadoSeeds* and a new land-leaf called *AvocadoSeedsTrees*. Thus, we need to modify `gcamreport` to include these items in the reporting dataset. In this example, the new items add to the final *cropland* area and several pollutant *emissions*, but do not have their own category in the template. diff --git a/vignettes/Reporting_Variables.Rmd b/vignettes/Reporting_Variables.Rmd index d448a854..6364cbb0 100644 --- a/vignettes/Reporting_Variables.Rmd +++ b/vignettes/Reporting_Variables.Rmd @@ -9,8 +9,8 @@ vignette: > The `gcamreport` package standardizes variables for reporting, which facilitates multi-model comparison projects. It follows the naming conventions, definitions, and units established by the [`openentrance`](https://github.com/openENTRANCE/openentrance) project, ensuring consistency across models. To compute global values for reported variables, the package applies different methodologies based on the characteristics of each variable. Below, you'll find a detailed list of all reported variables and their respective computational procedures. In the standardized datasets, these global values are categorized under the ``World'' region. -| Reported variable | World value computational procedure | -| ----------------------------------------------------------------------- | ------------------------------------------------------------------ | +| Reported variable | World value computational procedure | +| ----------------------------------------------------------------------- | ---------------------------------------------------------------------------------------- | | Agricultural Demand | Sum of regional values | | Agricultural Demand\|Crops | Sum of regional values | | Agricultural Demand\|Crops\|Energy | Sum of regional values | @@ -390,70 +390,70 @@ The `gcamreport` package standardizes variables for reporting, which facilitates | Land Cover\|Pasture | Sum of regional values | | Population | Sum of regional values | | Price\|Agriculture\|Crops\|Index | Weighted sum of regional values by commodity | -| Price\|Agriculture\|Crops\|Food\|Index | Mean of regional values | -| Price\|Agriculture\|Crops\|Food\|Corn\|Index | Mean of regional values | -| Price\|Agriculture\|Crops\|Food\|FiberCrop\|Index | Mean of regional values | -| Price\|Agriculture\|Crops\|Food\|Fruits\|Index | Mean of regional values | -| Price\|Agriculture\|Crops\|Food\|Legumes\|Index | Mean of regional values | -| Price\|Agriculture\|Crops\|Food\|MiscCrop\|Index | Mean of regional values | -| Price\|Agriculture\|Crops\|Food\|NutsSeeds\|Index | Mean of regional values | -| Price\|Agriculture\|Crops\|Food\|OilCrop\|Index | Mean of regional values | -| Price\|Agriculture\|Crops\|Food\|OilPalm\|Index | Mean of regional values | -| Price\|Agriculture\|Crops\|Food\|OtherGrain\|Index | Mean of regional values | -| Price\|Agriculture\|Crops\|Food\|Rice\|Index | Mean of regional values | -| Price\|Agriculture\|Crops\|Food\|RootTuber\|Index | Mean of regional values | -| Price\|Agriculture\|Crops\|Food\|Soybean\|Index | Mean of regional values | -| Price\|Agriculture\|Crops\|Food\|SugarCrop\|Index | Mean of regional values | -| Price\|Agriculture\|Crops\|Food\|Vegetables\|Index | Mean of regional values | -| Price\|Agriculture\|Crops\|Food\|Wheat\|Index | Mean of regional values | -| Price\|Agriculture\|Crops\|Feed\|Index | Mean of regional values | -| Price\|Agriculture\|Crops\|Energy\|Index | Mean of regional values | -| Price\|Agriculture\|Crops\|Other\|Index | Mean of regional values | -| Price\|Agriculture\|Livestock\|Food\|Index | Mean of regional values | -| Price\|Agriculture\|Livestock\|Food\|Beef\|Index | Mean of regional values | -| Price\|Agriculture\|Livestock\|Food\|Dairy\|Index | Mean of regional values | -| Price\|Agriculture\|Livestock\|Food\|Pork\|Index | Mean of regional values | -| Price\|Agriculture\|Livestock\|Food\|Poultry\|Index | Mean of regional values | -| Price\|Agriculture\|Livestock\|Food\|SheepGoat\|Index | Mean of regional values | -| Price\|Agriculture\|Livestock\|Other\|Index | Mean of regional values | +| Price\|Agriculture\|Crops\|Food\|Index | Weighted sum of regional values by commodity | +| Price\|Agriculture\|Crops\|Food\|Corn\|Index | Weighted sum of regional values by commodity | +| Price\|Agriculture\|Crops\|Food\|FiberCrop\|Index | Weighted sum of regional values by commodity | +| Price\|Agriculture\|Crops\|Food\|Fruits\|Index | Weighted sum of regional values by commodity | +| Price\|Agriculture\|Crops\|Food\|Legumes\|Index | Weighted sum of regional values by commodity | +| Price\|Agriculture\|Crops\|Food\|MiscCrop\|Index | Weighted sum of regional values by commodity | +| Price\|Agriculture\|Crops\|Food\|NutsSeeds\|Index | Weighted sum of regional values by commodity | +| Price\|Agriculture\|Crops\|Food\|OilCrop\|Index | Weighted sum of regional values by commodity | +| Price\|Agriculture\|Crops\|Food\|OilPalm\|Index | Weighted sum of regional values by commodity | +| Price\|Agriculture\|Crops\|Food\|OtherGrain\|Index | Weighted sum of regional values by commodity | +| Price\|Agriculture\|Crops\|Food\|Rice\|Index | Weighted sum of regional values by commodity | +| Price\|Agriculture\|Crops\|Food\|RootTuber\|Index | Weighted sum of regional values by commodity | +| Price\|Agriculture\|Crops\|Food\|Soybean\|Index | Weighted sum of regional values by commodity | +| Price\|Agriculture\|Crops\|Food\|SugarCrop\|Index | Weighted sum of regional values by commodity | +| Price\|Agriculture\|Crops\|Food\|Vegetables\|Index | Weighted sum of regional values by commodity | +| Price\|Agriculture\|Crops\|Food\|Wheat\|Index | Weighted sum of regional values by commodity | +| Price\|Agriculture\|Crops\|Feed\|Index | Weighted sum of regional values by commodity | +| Price\|Agriculture\|Crops\|Energy\|Index | Weighted sum of regional values by commodity | +| Price\|Agriculture\|Crops\|Other\|Index | Weighted sum of regional values by commodity | +| Price\|Agriculture\|Livestock\|Food\|Index | Weighted sum of regional values by commodity | +| Price\|Agriculture\|Livestock\|Food\|Beef\|Index | Weighted sum of regional values by commodity | +| Price\|Agriculture\|Livestock\|Food\|Dairy\|Index | Weighted sum of regional values by commodity | +| Price\|Agriculture\|Livestock\|Food\|Pork\|Index | Weighted sum of regional values by commodity | +| Price\|Agriculture\|Livestock\|Food\|Poultry\|Index | Weighted sum of regional values by commodity | +| Price\|Agriculture\|Livestock\|Food\|SheepGoat\|Index | Weighted sum of regional values by commodity | +| Price\|Agriculture\|Livestock\|Other\|Index | Weighted sum of regional values by commodity | | Price\|Carbon | Weighted sum of regional values by CO2 emissions | -| Price\|Carbon\|Energy | Sum of regional values | -| Price\|Carbon\|Energy\|Demand\|Industry | Sum of regional values | -| Price\|Carbon\|Energy\|Demand\|Residential and Commercial | Sum of regional values | -| Price\|Carbon\|Energy\|Demand\|Transportation | Sum of regional values | -| Price\|Carbon\|Energy\|Supply | Sum of regional values | -| Price\|Final Energy\|Buildings\|Residential | Mean of regional values | -| Price\|Final Energy\|Buildings\|Residential\|Electricity | Mean of regional values | -| Price\|Final Energy\|Buildings\|Residential\|Electricity\|Index | Mean of regional values | -| Price\|Final Energy\|Buildings\|Residential\|Gases\|Gas | Mean of regional values | -| Price\|Final Energy\|Buildings\|Residential\|Gases\|Gas\|Index | Mean of regional values | -| Price\|Final Energy\|Buildings\|Residential\|Liquids | Mean of regional values | -| Price\|Final Energy\|Buildings\|Residential\|Solids | Mean of regional values | -| Price\|Final Energy\|Buildings\|Residential\|Solids\|Biomass | Mean of regional values | -| Price\|Final Energy\|Buildings\|Residential\|Solids\|Coal | Mean of regional values | -| Price\|Final Energy\|Industry | Mean of regional values | -| Price\|Final Energy\|Industry\|Electricity | Mean of regional values | -| Price\|Final Energy\|Industry\|Gases | Mean of regional values | -| Price\|Final Energy\|Industry\|Liquids | Mean of regional values | -| Price\|Final Energy\|Industry\|Solids | Mean of regional values | -| Price\|Final Energy\|Industry\|Solids\|Biomass | Mean of regional values | -| Price\|Final Energy\|Industry\|Solids\|Coal | Mean of regional values | -| Price\|Final Energy\|Transportation | Mean of regional values | -| Price\|Final Energy\|Transportation\|Electricity | Mean of regional values | -| Price\|Final Energy\|Transportation\|Gases | Mean of regional values | -| Price\|Final Energy\|Transportation\|Liquids | Mean of regional values | -| Price\|Primary Energy | Mean of regional values | -| Price\|Primary Energy\|Biomass | Mean of regional values | -| Price\|Primary Energy\|Biomass\|Index | Mean of regional values | -| Price\|Primary Energy\|Coal | Mean of regional values | -| Price\|Primary Energy\|Coal\|Index | Mean of regional values | -| Price\|Primary Energy\|Gas | Mean of regional values | -| Price\|Primary Energy\|Gas\|Index | Mean of regional values | -| Price\|Primary Energy\|Oil | Mean of regional values | -| Price\|Primary Energy\|Oil\|Index | Mean of regional values | -| Price\|Secondary Energy | Mean of regional values | -| Price\|Secondary Energy\|Electricity | Mean of regional values | -| Price\|Secondary Energy\|Electricity\|Index | Mean of regional values | +| Price\|Carbon\|Energy | Weighted sum of regional values by CO2 emissions | +| Price\|Carbon\|Energy\|Demand\|Industry | Weighted sum of regional values by CO2 emissions | +| Price\|Carbon\|Energy\|Demand\|Residential and Commercial | Weighted sum of regional values by CO2 emissions | +| Price\|Carbon\|Energy\|Demand\|Transportation | Weighted sum of regional values by CO2 emissions | +| Price\|Carbon\|Energy\|Supply | Weighted sum of regional values by CO2 emissions | +| Price\|Final Energy\|Buildings\|Residential | Weighted sum of regional values by commodity | +| Price\|Final Energy\|Buildings\|Residential\|Electricity | Weighted sum of regional values by commodity | +| Price\|Final Energy\|Buildings\|Residential\|Electricity\|Index | Weighted sum of regional values by commodity | +| Price\|Final Energy\|Buildings\|Residential\|Gases\|Gas | Weighted sum of regional values by commodity | +| Price\|Final Energy\|Buildings\|Residential\|Gases\|Gas\|Index | Weighted sum of regional values by commodity | +| Price\|Final Energy\|Buildings\|Residential\|Liquids | Weighted sum of regional values by commodity | +| Price\|Final Energy\|Buildings\|Residential\|Solids | Weighted sum of regional values by commodity | +| Price\|Final Energy\|Buildings\|Residential\|Solids\|Biomass | Weighted sum of regional values by commodity | +| Price\|Final Energy\|Buildings\|Residential\|Solids\|Coal | Weighted sum of regional values by commodity | +| Price\|Final Energy\|Industry | Weighted sum of regional values by commodity | +| Price\|Final Energy\|Industry\|Electricity | Weighted sum of regional values by commodity | +| Price\|Final Energy\|Industry\|Gases | Weighted sum of regional values by commodity | +| Price\|Final Energy\|Industry\|Liquids | Weighted sum of regional values by commodity | +| Price\|Final Energy\|Industry\|Solids | Weighted sum of regional values by commodity | +| Price\|Final Energy\|Industry\|Solids\|Biomass | Weighted sum of regional values by commodity | +| Price\|Final Energy\|Industry\|Solids\|Coal | Weighted sum of regional values by commodity | +| Price\|Final Energy\|Transportation | Weighted sum of regional values by commodity | +| Price\|Final Energy\|Transportation\|Electricity | Weighted sum of regional values by commodity | +| Price\|Final Energy\|Transportation\|Gases | Weighted sum of regional values by commodity | +| Price\|Final Energy\|Transportation\|Liquids | Weighted sum of regional values by commodity | +| Price\|Primary Energy | Weighted sum of regional values by commodity | +| Price\|Primary Energy\|Biomass | Weighted sum of regional values by commodity | +| Price\|Primary Energy\|Biomass\|Index | Weighted sum of regional values by commodity | +| Price\|Primary Energy\|Coal | Weighted sum of regional values by commodity | +| Price\|Primary Energy\|Coal\|Index | Weighted sum of regional values by commodity | +| Price\|Primary Energy\|Gas | Weighted sum of regional values by commodity | +| Price\|Primary Energy\|Gas\|Index | Weighted sum of regional values by commodity | +| Price\|Primary Energy\|Oil | Weighted sum of regional values by commodity | +| Price\|Primary Energy\|Oil\|Index | Weighted sum of regional values by commodity | +| Price\|Secondary Energy | Weighted sum of regional values by commodity | +| Price\|Secondary Energy\|Electricity | Weighted sum of regional values by commodity | +| Price\|Secondary Energy\|Electricity\|Index | Weighted sum of regional values by commodity | | Primary Energy | Sum of regional values | | Primary Energy\|Biomass | Sum of regional values | | Primary Energy\|Biomass\|1st Generation | Sum of regional values |