Update section on using multiple files

Author: shireen122

Intro_DHSdata_Analysis/4_Using_Multiple_Files/Readme.md

@@ -0,0 +1,63 @@
+# Using Multiple Files
+
+When analyzing DHS data, you will sometimes need to use more than one file. 
+For instance, some information about the household (such as water and sanitation) can only be found in the HR or PR file. 
+However, you may need this information for an analysis of women so you would need to merge to the IR file.
+
+For more detailed explanation on merging please check the DHS Program website for this topic [Merging DHS data](https://www.dhsprogram.com/data/Merging-Datasets.cfm)
+
+There are several types of merges that can be performed. The code for the most common types are available here.
+When possible, the DHS model datasets are used.
+
+You may also need to use multiple files when performing a multi-country analysis or trend analysis.
+This involves appending or pooling two or more surveys of the same file type. 
+Appending can also be used to append data from men (MR) and women (IR) which is shown in the code for merging AR to IR or MR files. 
+
+### mergePR_toKRorIR
+The code will show how to merge a PR to a KR file, and a PR to an IR file.
+
+### mergeHR_toIR
+The purpose of this code is to show an example of a many to one merge by merging the HR file to the IR file.
+
+### mergeAR_toIRorMR
+This shows how to merge the HIV test results from the AR file to the IR file or the MR file.
+The code also shows how to append the merged IR and MR files to a dataset that contains both men and women.
+
+### mergeHW
+Older datasets may not contain the anthropometry variables according to the WHO reference. 
+HW files are created with the WHO reference nutrition indicators that can be merged to datasets.
+Here we show an example using the Bangladesh 1999-2000 survey since there is no HW file for model datasets.
+
+### mergeWI 
+Older datasets sometimes do not contain the wealth index. 
+To include the wealth index, a WI file was created for these surveys. 
+The Bangladesh 1999-2000 survey as an example since there is no WI file for model datasets.
+The code will show how to merge the WI with an IR file and notes on how to merge with MR or PR/HR files. 
+
+### mergeGPS
+This shows how to merge an IR file with the GC (points) file in STATA
+The Nigeria 2015 MIS data is used for this example, but the code should work for any DHS survey
+
+### mergeHHhead_toIR
+Construct a file that attaches the characteristics of the household head to each eligible woman in the household.
+
+### mergeChild01_17_parents
+Merging children 0-17 in the PR file with all the data about their mothers and fathers and all the data in the BR file.
+This will produce three different data files. See description in the file for more detail.
+The execution begins at the bottom of the do file where you need to specify your paths. 
+
+### mergeIR_toSR
+This is an example of how to produce an IRSR file, using separate IR and SR files. 
+The Cote d'Ivoire 2022 survey is used since the model datasets do not have an SR file. 
+
+### merge_loop
+This code shows how to perform a merge in a loop in the case you need to perform merges for several countries. 
+
+### pool_trends
+This shows how to append two surveys from one country for a trend analysis. 
+The Nigeria 2013 and 2018 DHS are used as an example for one country and code is provided to show how an append of several countries can be performed in a loop.
+
+
+
+##### author: Shireen Assaf
+##### last updated: July 26, 2023 by Shireen Assaf

…hildren_mothers_fathers_do_14Feb2023.txt …ultiple_Files/mergeChild01_17_parents.doIntro_DHSdata_Analysis/4_Using_Multiple_Files/merge_children_mothers_fathers_do_14Feb2023.txt renamed to Intro_DHSdata_Analysis/4_Using_Multiple_Files/mergeChild01_17_parents.do Intro_DHSdata_Analysis/4_Using_Multiple_Files/merge_children_mothers_fathers_do_14Feb2023.txt renamed to Intro_DHSdata_Analysis/4_Using_Multiple_Files/mergeChild01_17_parents.do

@@ -1,9 +1,12 @@
-* do e:\DHS\programs\merging\merge_children_mothers_fathers_do_14Feb2023.txt
-
-/*
- Program to merge children 0-17 in the PR file with ALL the data about their
+/*****************************************************************************************************
+Program: 			mergeChild01_17_parents.do
+Purpose: 			Program to merge children 0-17 in the PR file with ALL the data about their
    mothers and fathers and ALL the data in the BR file
+Author:				Tom Pullum
+Date last modified: Feb 14, 2023 by Tom Pullum
+*****************************************************************************************************/
 
+/* Description: 
 Here are the category labels for "Mother's status", with more explanation:
 0 "Dead"                   hv111==0
 1 "Not in HH"              (hv111>0 & hv112==0) | hv112==. 
@@ -272,18 +275,18 @@ end
 ******************************************************************************
 * Execution begins here
 
-* Specify workspace
-cd e:\DHS\DHS_data\scratch
+* Specify your workspace, change to your own path
+*cd "C:\.."
 
-* Specify path to the data
-scalar spath="C:\Users\26216\ICF\Analysis - Shared Resources\Data\DHSdata"
+* Specify path to the data, change to your own
+scalar spath="C:\Data"
 
 * specify the first six characters of the PR, IR, MR, and BR file names.
 * Note that characters 5-6 may not be the same in all four files.
-scalar sfn_PR="RWPR61"
-scalar sfn_IR="RWIR61"
-scalar sfn_MR="RWMR61"
-scalar sfn_BR="RWBR61"
+scalar sfn_PR="ZZPR62"
+scalar sfn_IR="ZZIR62"
+scalar sfn_MR="ZZMR61"
+scalar sfn_BR="ZZBR62"
 
 make_workfiles
 prepare_workfiles_for_merges
@@ -292,10 +295,17 @@ merge_with_IR
 merge_with_MR
 merge_with_BR
 
-
-
-
-
-
-
+*erase all temporary files
+erase BR.dta
+erase BRtemp.dta
+erase IR.dta
+erase IRtemp.dta
+erase MR.dta
+erase MRtemp.dta
+erase PR.dta
+erase PRtemp_ch.dta
+erase PRtemp_ch_mo.dta
+erase PRtemp_ch_mo_fa.dta
+erase PRtemp_fa.dta
+erase PRtemp_mo.dta
 

…ysis/4_Using_Multiple_Files/GPS_Merge.do …lysis/4_Using_Multiple_Files/mergeGPS.doIntro_DHSdata_Analysis/4_Using_Multiple_Files/GPS_Merge.do renamed to Intro_DHSdata_Analysis/4_Using_Multiple_Files/mergeGPS.do Intro_DHSdata_Analysis/4_Using_Multiple_Files/GPS_Merge.do renamed to Intro_DHSdata_Analysis/4_Using_Multiple_Files/mergeGPS.do

@@ -3,22 +3,15 @@
 * Created by: Tom Fish <gpsrequests@dhsprogram.com>
 * Created: 30 October 2018
 * Updated: 06 April 2021
-* Purpose: Explan how to merge an IR file with the GC (points) file in STATA
-* Notes: The Nigeria data is used for this example, but the code should work for any DHS survey
+* Purpose: Explain how to merge an IR file with the GC (points) file in STATA
+* Notes: The Nigeria 2015 MIS data is used for this example, but the code should work for any DHS survey
 ******************************************************************************************/
 clear all
 set more off
 
-* Set folders that I am working on
-local ptsDir "C:\Data\DHSdata\GPS\"
-local dataDir "C:\Data\DHSdata\Recode\"
-local working "C:\Working\"
-
-* Make sure we are working in our working directory
-cd "`working'"
 
 * Convert the shapefile into a dta file to merge in STATA
-shp2dta using "`ptsDir'NGGE71FL.shp", database(ngpts) coordinates(ngcoord) genid(id)
+shp2dta using "NGGE71FL.shp", database(ngpts) coordinates(ngcoord) genid(id)
 
 * Open up the table portion of the shapefile
 use ngpts
@@ -31,9 +24,9 @@ sort v001
 save ngpts, replace
 
 * Open the IR file
-use "`dataDir'NGIR71FL.DTA", clear
+use "NGIR71FL.DTA", clear
 
-* Do a 1 to Many merge/join by joining the GPS points (ngpts.dta) to the IR file
+* Do a 1 to many merge/join by joining the GPS points (ngpts.dta) to the IR file
 sort v001
 merge v001 using ngpts.dta
 
@@ -42,4 +35,4 @@ tab _merge
 drop _merge id
 
 * Save the merged file
-save NG_Merged, replace
+save NG_merged, replace

Intro_DHSdata_Analysis/4_Using_Multiple_Files/mergeHHhead_toIR.do

@@ -0,0 +1,38 @@
+/*****************************************************************************************************
+Program: 			mergeHHhead_toIR.do
+Purpose: 			Construct a file that attaches the characteristics of the household head to each eligible woman in the household.
+Author:				Tom Pullum
+Date last modified: June 16, 2023 by Tom Pullum
+*****************************************************************************************************/
+
+use "ZZPR62FL.DTA", clear 
+
+* reduce to just household heads
+keep if hv101==1
+
+* select variables you need, not necessarily as illustrated here
+keep hvidx hv0* hv1*
+
+* rename to avoid possible confusion
+rename hv* head_hv*
+
+* specify variables for the merge
+rename head_hv001 cluster 
+rename head_hv002 hh
+sort cluster hh
+save temp.dta, replace
+
+* open IR file
+use "ZZIR62FL.DTA", clear 
+rename v001 cluster
+rename v002 hh
+sort cluster hh
+
+* merge is m:1 because there may be more than 1 eligible woman per household
+merge m:1 cluster hh using temp.dta
+tab _merge
+
+* _merge=2 for households that do not include any eligible women
+drop if _merege==2
+
+drop _merge

Intro_DHSdata_Analysis/4_Using_Multiple_Files/mergeIR_toSR.do

@@ -0,0 +1,39 @@
+/*****************************************************************************************************
+Program: 			mergeIR_toSR.do
+Purpose: 			This is an example of how to produce an IRSR file, using separate IR and SR files. 
+Author:				Tom Pullum
+Date last modified: May 26, 2023 by Tom Pullum
+*****************************************************************************************************/
+
+* This is an example of how to produce an IRSR file, using separate IR and SR files. 
+
+* This is useful for calculating adult and maternal mortality rates with a program 
+* designed for surveys in which the sibling histories are in the IR file.
+
+* Other modifications may be needed.
+* Illustrated with preliminary IR and SR files from Cote d'Ivoire 2022 
+* The new file is saved with "IRSR" in the filename
+
+* path to your workding directory where your data files should also be stored. 
+cd "C:\.."
+
+use CIIR80FL.dta, clear
+sort v001 v002 v003
+save temp.dta, replace
+
+use CISR80FL.dta, clear
+
+keep v001 v002 v003  mm*
+rename mm* mm*_
+rename mmidx_ index
+quietly summarize index
+local lmax_index=r(max)
+reshape wide mm*_ , i(v001 v002 v003) j(index)
+merge v001 v002 v003 using temp.dta
+
+tab _merge
+drop _merge
+* _merge=2 for women who did not report any siblings; they must be kept in the data.
+* mmidx_* is still in the data.
+
+save CIIRSR80FL.dta, replace

Intro_DHSdata_Analysis/4_Using_Multiple_Files/mergeWI.do

@@ -4,7 +4,7 @@ Purpose: 			example shown to merge WR to IR file. Notes given on how to merge wi
 Author:				Shireen Assaf
 Date last modified: May 27, 2021 by Shireen Assaf 
 
-Notes:				Older datasets can sometimes not contain the wealth index. To include the wealth index a WI file was created for these surveys. Will use the Bangladesh 1999-2000 survey as an example since model datasets do not have an WI file.
+Notes:				Older datasets sometimes do not contain the wealth index. To include the wealth index, a WI file was created for these surveys. Will use the Bangladesh 1999-2000 survey as an example since model datasets do not have an WI file.
 					
 					Notes are given at the end of the file on how to perform other merges of WI file with other file types. 
 *****************************************************************************************************/

Intro_DHSdata_Analysis/4_Using_Multiple_Files/merge_loop.do

@@ -0,0 +1,46 @@
+/*****************************************************************************************************
+Program: 			merge_loop.do
+Purpose: 			Merge coded KR files to HR files in a loop for several countries
+Author:				Shireen Assaf			
+Date last modified: July 26, 2023 by Shireen Assaf 
+Notes:				This assumes that you have HR and KR coded files from several countries saved for example as CDHRcoded.dta and CDKRcoded.dta. The master file in the case is the KR file. The same logic can be used for other merges. 
+*****************************************************************************************************/
+
+* list of HR surveys from 6 countries
+*CDHR61FL ETHR71FL KEHR72FL NGHR7BFL TZHR7BFL UGHR7BFL
+
+* list of KR surveys from the same 6 countries
+global krdata "CDKR61FL ETKR71FL KEKR72FL NGKR7BFL TZKR7BFL UGKR7BFL"
+
+*list your HR files in this format 
+tokenize "CDHR ETHR KEHR NGHR TZHR UGHR"
+
+*being loop
+foreach c in $krdata {
+
+*getting country two letter acronym. See notes, the assumption here is that you have KR and HR coded files. 	
+local cn=substr("`c'",1,2)
+
+use "`cn'KRcoded.dta", clear
+
+*perform a many to one merge since you are merging household data to KR data (many children in one household).
+*this merge would be 1:1 if it was PR to KR merge for instance. See code for other merges in this section.
+* the local `1' here refers to the first survey in the tokenize list. 
+
+merge m:1 v001 v002 using "`1'coded.dta"
+
+*renaming the _merge created by the merge in case you want to perform other merges. 
+rename _merge KRmerge
+
+* keep only matched cases
+keep if KRmerge==3
+
+*save merged file	
+save "`cn'HRKRmerged.dta", replace
+
+* this shifts to the next survey in the tokensize list and the loop continues until the list is completed. 
+mac shift
+
+  }
+  
+  

Intro_DHSdata_Analysis/4_Using_Multiple_Files/pool_trends.do

@@ -0,0 +1,66 @@
+/*****************************************************************************************************
+Program: 			pool_trends.do
+Purpose: 			This is an example of how to append two surveys from different years for the same country for a trend analysis. We use DHS data from Nigeria 2013 and 2018 as an example and also show how to append in a loop for several countries. See notes below. 
+Author:				Shireen Assaf
+Date last modified: July 26, 2023 by Shireen Assaf
+*****************************************************************************************************/
+
+* Append two surveys from the same country
+	
+use NGIR6AFL.dta, clear  // open first survey
+gen yr=1 // create a year variable
+	
+append using NGIR7BFL.dta // append using second survey
+replace yr=2 if yr==. // replace values for second survey
+	
+label var yr "year of survey"
+label define yr 1 2013 2 2018
+label  values yr yr
+tab yr
+	
+* can also just check v007
+tab v007
+
+gen wt=v005/1000000
+
+*run programs to code country-specific strata
+* Use the Survey_strata.do file from section 2
+quietly do "Survey_strata.do"
+	
+* svy set for combined data
+egen strata2=group(yr strata)
+egen v021r = group(yr v021)
+svyset v021r [pw=wt], strata(strata2) singleunit(centered)
+	
+save NGappend.dta, replace // save appended file
+
+	
+*********************************************
+
+* how to run a loop for appending files for trend analysis.
+* the code will append each two sets of surveys for each country.
+
+* create two globals for each survey list. The irdata1 global is the most recent survey that will be appending to irdata2 which is an earlier survey for each country. The list of countries should be in the same order for both globals. 
+
+global irdata1 "BDIR7RFL BJIR71FL CMIR71FL KEIR72FL LBIR7AFL RWIR81FL SLIR7AFL UGIR7BFL ZMIR71FL ZWIR72FL"
+global irdata2 "BDIR51FL BJIR51FL CMIR44FL KEIR42FL LBIR51FL RWIR53FL SLIR51FL UGIR52FL ZMIR51FL ZWIR52FL"
+
+
+tokenize $irdata1
+
+foreach c in $irdata2 {
+use "`1'coded.dta", clear
+gen yr=1
+
+append using "`c'coded.dta"
+replace yr=2 if yr==.   
+
+local cn = substr(v000,1,2)
+save "`cn'append.dta", replace
+
+mac shift 1
+}
+
+*/
+
+