harmonize Xport input data, improve gdx_ref fixing

[orichters]

Purpose of this PR

• close exports not correctly fixed to reference gdx #1405
• this code in core/datainput.gms that adjusts the historic trade patterns was responsible for an inconsistencies between trade values in the scenario and the reference scenarios for t < cm_startyear.
• pm_IO_trade was loaded for ttot, but the rescaling only happened for t
• By adjusting this also ttot, the problem is fixed, also leading to harmonized trade patterns in 2000 for the NPi run.

Results

cd /p/tmp/oliverr/remind/output
diff -y <(dumpgdx SSP2EU-NPi_2023-09-14_19.08.39/fulldata.gdx pm_IO_trade 20[0-5][0-5],.*,) <( dumpgdx SSP2EU-NDC_2023-09-14_21.55.19/fulldata.gdx pm_IO_trade 20[0-5][0-5],.*,)  | grep "|"

shows no difference anymore.

Different to before the fix:

diff -y <(dumpgdx SSP2EU-NPi_2023-09-13_21.30.30/fulldata.gdx pm_IO_trade 20[0-5][0-5],.*,) <( dumpgdx SSP2EU-NDC_2023-09-14_00.14.50/fulldata.gdx pm_IO_trade 20[0-5][0-5],.*,)  | grep "|"
2005  LAM  peoil   Xport  0.50783192091295                    | 2005  LAM  peoil   Xport  0.5183692715481
2005  LAM  pegas   Xport  0.0336791276091469                  | 2005  LAM  pegas   Xport  0.03453333156
2005  LAM  pecoal  Xport  0.0541742707446877                  | 2005  LAM  pecoal  Xport  0.0560003886057
2005  OAS  peoil   Xport  0.265219809182672                   | 2005  OAS  peoil   Xport  0.2707230357615
2005  OAS  pegas   Xport  0.101792577506475                   | 2005  OAS  pegas   Xport  0.104374343367
2005  OAS  pecoal  Xport  0.115078840265428                   | 2005  OAS  pecoal  Xport  0.1189579423326
2005  SSA  peoil   Xport  0.331829840742                      | 2005  SSA  peoil   Xport  0.3387152042631
2005  SSA  pegas   Xport  0.016329188806252                   | 2005  SSA  pegas   Xport  0.016743346137
2005  SSA  pecoal  Xport  0.0619557156124171                  | 2005  SSA  pecoal  Xport  0.064044132075
2005  EUR  peoil   Xport  0.464322439110551                   | 2005  EUR  peoil   Xport  0.4739569818543
2005  EUR  pegas   Xport  0.0902252208651719                  | 2005  EUR  pegas   Xport  0.0925136037777
2005  EUR  pecoal  Xport  0.0304261459214709                  | 2005  EUR  pecoal  Xport  0.031451756931
2005  NEU  peoil   Xport  0.182403294360543                   | 2005  NEU  peoil   Xport  0.1861881046305
2005  NEU  pegas   Xport  0.0918903228149865                  | 2005  NEU  pegas   Xport  0.09422093772
2005  NEU  pecoal  Xport  0.00193097579845831                 | 2005  NEU  pecoal  Xport  0.0019960655421
2005  MEA  peoil   Xport  1.60259512038512                    | 2005  MEA  peoil   Xport  1.6358484587718
2005  MEA  pegas   Xport  0.153774620390542                   | 2005  MEA  pegas   Xport  0.157674807171
2005  MEA  pecoal  Xport  0.000493716820035473                | 2005  MEA  pecoal  Xport  0.0005103591318
2005  REF  peoil   Xport  0.574503907137014                   | 2005  REF  peoil   Xport  0.586424680254
2005  REF  pegas   Xport  0.297717737752667                   | 2005  REF  pegas   Xport  0.305268754833
2005  REF  pecoal  Xport  0.0882049177480886                  | 2005  REF  pecoal  Xport  0.0911781479091
2005  CAZ  peoil   Xport  0.163874894046517                   | 2005  CAZ  peoil   Xport  0.1672752459105
2005  CAZ  pegas   Xport  0.129200413526387                   | 2005  CAZ  pegas   Xport  0.132477324525
2005  CAZ  pecoal  Xport  0.217843827656906                   | 2005  CAZ  pecoal  Xport  0.2251869538149
2005  CHA  peoil   Xport  0.0563017576972989                  | 2005  CHA  peoil   Xport  0.0574700012397
2005  CHA  pegas   Xport  0.00321711318950684                 | 2005  CHA  pegas   Xport  0.003298708854
2005  CHA  pecoal  Xport  0.0711154240505526                  | 2005  CHA  pecoal  Xport  0.0735125979168
2005  IND  peoil   Xport  0.0315164941794959                  | 2005  IND  peoil   Xport  0.032170451397
2005  IND  pecoal  Xport  0.0011160110393988                  | 2005  IND  pecoal  Xport  0.0011536297773
2005  JPN  peoil   Xport  0.0116929113646988                  | 2005  JPN  peoil   Xport  0.0119355355518
2005  JPN  pecoal  Xport  0.00151379122378952                 | 2005  JPN  pecoal  Xport  0.0015648184209
2005  USA  peoil   Xport  0.0754624231452459                  | 2005  USA  peoil   Xport  0.0770282443938
2005  USA  pegas   Xport  0.0216155793925636                  | 2005  USA  pegas   Xport  0.022163815485
2005  USA  pecoal  Xport  0.0378153617279673                  | 2005  USA  pecoal  Xport  0.0390900499983
2010  LAM  peoil   Xport  0.474037429794855                   | 2010  LAM  peoil   Xport  0.465533136957
2010  LAM  pegas   Xport  0.0423509649373783                  | 2010  LAM  pegas   Xport  0.042210094248
2010  LAM  pecoal  Xport  0.0649945254588924                  | 2010  LAM  pecoal  Xport  0.0658477616769
2010  OAS  peoil   Xport  0.297620947472433                   | 2010  OAS  peoil   Xport  0.2922815891583
2010  OAS  pegas   Xport  0.11080019555108                    | 2010  OAS  pegas   Xport  0.1104316443279
2010  OAS  pecoal  Xport  0.238958231406731                   | 2010  OAS  pecoal  Xport  0.2420952312723
2010  SSA  peoil   Xport  0.373032345798636                   | 2010  SSA  peoil   Xport  0.3663400972392
2010  SSA  pegas   Xport  0.0325315091358268                  | 2010  SSA  pegas   Xport  0.032423300595
2010  SSA  pecoal  Xport  0.059585584613755                   | 2010  SSA  pecoal  Xport  0.0603678132477
2010  EUR  peoil   Xport  0.483041328329502                   | 2010  EUR  peoil   Xport  0.4743755043867
2010  EUR  pegas   Xport  0.116216231630105                   | 2010  EUR  pegas   Xport  0.1158296652156
2010  EUR  pecoal  Xport  0.0277943088575674                  | 2010  EUR  pecoal  Xport  0.0281591874501
2010  NEU  peoil   Xport  0.136243900242492                   | 2010  NEU  peoil   Xport  0.1337996670402
2010  NEU  pegas   Xport  0.11857346441716                    | 2010  NEU  pegas   Xport  0.118179057213
2010  NEU  pecoal  Xport  0.00219013870420055                 | 2010  NEU  pecoal  Xport  0.0022188904437
2010  MEA  peoil   Xport  1.54287867659372                    | 2010  MEA  peoil   Xport  1.5151992334647
2010  MEA  pegas   Xport  0.240876596539294                   | 2010  MEA  pegas   Xport  0.240075376254
2010  MEA  pecoal  Xport  0.000321097266831838                | 2010  MEA  pecoal  Xport  0.0003253125729
2010  REF  peoil   Xport  0.684927081229682                   | 2010  REF  peoil   Xport  0.6726394007529
2010  REF  pegas   Xport  0.25796422552988                    | 2010  REF  pegas   Xport  0.257106167199
2010  REF  pecoal  Xport  0.123640818117481                   | 2010  REF  pecoal  Xport  0.1252639521168
2010  CAZ  peoil   Xport  0.195425439967816                   | 2010  CAZ  peoil   Xport  0.1919194822839
2010  CAZ  pegas   Xport  0.13330492454198                    | 2010  CAZ  pegas   Xport  0.132861516543
2010  CAZ  pecoal  Xport  0.278137221624687                   | 2010  CAZ  pecoal  Xport  0.2817885560931
2010  CHA  peoil   Xport  0.0688778823813202                  | 2010  CHA  peoil   Xport  0.0676422042576
2010  CHA  pegas   Xport  0.00504339446628978                 | 2010  CHA  pegas   Xport  0.0050266187811
2010  CHA  pecoal  Xport  0.0191638738349554                  | 2010  CHA  pecoal  Xport  0.0194154536583
2010  IND  peoil   Xport  0.0830530327435899                  | 2010  IND  peoil   Xport  0.0815630505879
2010  IND  pecoal  Xport  0.00158704927982928                 | 2010  IND  pecoal  Xport  0.0016078837719
2010  JPN  peoil   Xport  0.0236029040584678                  | 2010  JPN  peoil   Xport  0.0231794649051
2010  JPN  pecoal  Xport  0.00061824476691207                 | 2010  JPN  pecoal  Xport  0.0006263609709
2010  USA  peoil   Xport  0.155366133826484                   | 2010  USA  peoil   Xport  0.152578845279
2010  USA  pegas   Xport  0.034837253095807                   | 2010  USA  pegas   Xport  0.034721375031
2010  USA  pecoal  Xport  0.0631942005937577                  | 2010  USA  pecoal  Xport  0.0640238024769
2015  LAM  peoil   Xport  0.442087437592082                   | 2015  LAM  peoil   Xport  0.4474816997301
2015  LAM  pegas   Xport  0.0446024650310624                  | 2015  LAM  pegas   Xport  0.045904382181
2015  LAM  pecoal  Xport  0.0670877273547272                  | 2015  LAM  pecoal  Xport  0.0689399249091
2015  OAS  peoil   Xport  0.315883374113339                   | 2015  OAS  peoil   Xport  0.3197377196118
2015  OAS  pegas   Xport  0.104681641220489                   | 2015  OAS  pegas   Xport  0.1077372307242
2015  OAS  pecoal  Xport  0.299257556740466                   | 2015  OAS  pecoal  Xport  0.3075196359102
2015  SSA  peoil   Xport  0.352950593053986                   | 2015  SSA  peoil   Xport  0.3572572253145
2015  SSA  pegas   Xport  0.0379103441527224                  | 2015  SSA  pegas   Xport  0.039016922616
2015  SSA  pecoal  Xport  0.0689540322552309                  | 2015  SSA  pecoal  Xport  0.0708577558563
2015  EUR  peoil   Xport  0.525192534256202                   | 2015  EUR  peoil   Xport  0.5316008280954
2015  EUR  pegas   Xport  0.125511053281096                   | 2015  EUR  pegas   Xport  0.1291746398712
2015  EUR  pecoal  Xport  0.0224678058589459                  | 2015  EUR  pecoal  Xport  0.0230881102977
2015  NEU  peoil   Xport  0.129007193987866                   | 2015  NEU  peoil   Xport  0.130581313863
2015  NEU  pegas   Xport  0.125466750021434                   | 2015  NEU  pegas   Xport  0.129129043428
2015  NEU  pecoal  Xport  0.00163546105429333                 | 2015  NEU  pecoal  Xport  0.0016806138279
2015  MEA  peoil   Xport  1.58540809318283                    | 2015  MEA  peoil   Xport  1.6047529243704
2015  MEA  pegas   Xport  0.2290963360999                     | 2015  MEA  pegas   Xport  0.2357835102
2015  MEA  pecoal  Xport  0.000234590013805423                | 2015  MEA  pecoal  Xport  0.0002410667133
2015  REF  peoil   Xport  0.674556812822741                   | 2015  REF  peoil   Xport  0.6827876195952
2015  REF  pegas   Xport  0.321030059432128                   | 2015  REF  pegas   Xport  0.3304007195454
2015  REF  pecoal  Xport  0.146951429728308                   | 2015  REF  pecoal  Xport  0.1510085514924
2015  CAZ  peoil   Xport  0.264061360820571                   | 2015  CAZ  peoil   Xport  0.2672833845192
2015  CAZ  pegas   Xport  0.122584896802783                   | 2015  CAZ  pegas   Xport  0.126163070775
2015  CAZ  pecoal  Xport  0.353049035519428                   | 2015  CAZ  pecoal  Xport  0.3627962215689
2015  CHA  peoil   Xport  0.0818814098765738                  | 2015  CHA  peoil   Xport  0.0828805103973
2015  CHA  pegas   Xport  0.00439201229818225                 | 2015  CHA  pegas   Xport  0.0045202123008
2015  CHA  pecoal  Xport  0.012615132973722                   | 2015  CHA  pecoal  Xport  0.012963419007
2015  IND  peoil   Xport  0.08557383482157                    | 2015  IND  peoil   Xport  0.08661798957
2015  IND  pecoal  Xport  0.000933547741906657                | 2015  IND  pecoal  Xport  0.0009593216787
2015  JPN  peoil   Xport  0.0242117483971142                  | 2015  JPN  peoil   Xport  0.0245071752891
2015  JPN  pecoal  Xport  0.000711535161792                   | 2015  JPN  pecoal  Xport  0.0007311796443
2015  USA  peoil   Xport  0.284466582359323                   | 2015  USA  peoil   Xport  0.2879375864736
2015  USA  pegas   Xport  0.0525017382006025                  | 2015  USA  pegas   Xport  0.054034230033
2015  USA  pecoal  Xport  0.0568017145271746                  | 2015  USA  pecoal  Xport  0.0583699297713
2020  LAM  peoil   Xport  0.343966278806212                   | 2020  LAM  peoil   Xport  0.3473435878563
2020  LAM  pegas   Xport  0.0357994728552768                  | 2020  LAM  pegas   Xport  0.0368479760565
2020  LAM  pecoal  Xport  0.0612603699572091                  | 2020  LAM  pecoal  Xport  0.064771863891
2020  OAS  peoil   Xport  0.277227751374385                   | 2020  OAS  peoil   Xport  0.2799497734194
2020  OAS  pegas   Xport  0.0862191912314832                  | 2020  OAS  pegas   Xport  0.08874439875
2020  OAS  pecoal  Xport  0.302694612413238                   | 2020  OAS  pecoal  Xport  0.3200453123196
2020  SSA  peoil   Xport  0.269837522437629                   | 2020  SSA  peoil   Xport  0.2724869818839
2020  SSA  pegas   Xport  0.0455954123147807                  | 2020  SSA  pegas   Xport  0.0469308212457
2020  SSA  pecoal  Xport  0.0678081422526832                  | 2020  SSA  pecoal  Xport  0.0716949597882
2020  EUR  peoil   Xport  0.46505035964429                    | 2020  EUR  peoil   Xport  0.4696165595457
2020  EUR  pegas   Xport  0.0829650438146591                  | 2020  EUR  pegas   Xport  0.0853949431146
2020  EUR  pecoal  Xport  0.0168461702441946                  | 2020  EUR  pecoal  Xport  0.0178118063424
2020  NEU  peoil   Xport  0.136776571178622                   | 2020  NEU  peoil   Xport  0.1381195422201
2020  NEU  pegas   Xport  0.1211812227702                     | 2020  NEU  pegas   Xport  0.1247304063159
2020  NEU  pecoal  Xport  0.000723701650400052                | 2020  NEU  pecoal  Xport  0.0007651848141
2020  MEA  peoil   Xport  1.49043458558203                    | 2020  MEA  peoil   Xport  1.5050687474886
2020  MEA  pegas   Xport  0.236172457989487                   | 2020  MEA  pegas   Xport  0.2430895313007
2020  MEA  pecoal  Xport  0.000411436425549008                | 2020  MEA  pecoal  Xport  0.0004350202941
2020  REF  peoil   Xport  0.630034330721911                   | 2020  REF  peoil   Xport  0.6362204622648
2020  REF  pegas   Xport  0.356593823390906                   | 2020  REF  pegas   Xport  0.3670378253703
2020  REF  pecoal  Xport  0.186214046154793                   | 2020  REF  pecoal  Xport  0.1968879858309
2020  CAZ  peoil   Xport  0.312973725494329                   | 2020  CAZ  peoil   Xport  0.3160467272991
2020  CAZ  pegas   Xport  0.197080722434563                   | 2020  CAZ  pegas   Xport  0.2028528679968
2020  CAZ  pecoal  Xport  0.339880129492851                   | 2020  CAZ  pecoal  Xport  0.359362333302
2020  CHA  peoil   Xport  0.107466544143838                   | 2020  CHA  peoil   Xport  0.1085217281967
2020  CHA  pegas   Xport  0.00611285062281745                 | 2020  CHA  pegas   Xport  0.0062918851989
2020  CHA  pecoal  Xport  0.00556029530313153                 | 2020  CHA  pecoal  Xport  0.0058790159253
2020  IND  peoil   Xport  0.0801795314417063                  | 2020  IND  peoil   Xport  0.08096679192
2020  IND  pecoal  Xport  0.00102360522711634                 | 2020  IND  pecoal  Xport  0.0010822791063
2020  JPN  peoil   Xport  0.0126492358785455                  | 2020  JPN  peoil   Xport  0.0127734352011
2020  JPN  pecoal  Xport  0.00283107864357695                 | 2020  JPN  pecoal  Xport  0.0029933583603
2020  USA  peoil   Xport  0.506324566740804                   | 2020  USA  peoil   Xport  0.5112960265812
2020  USA  pegas   Xport  0.160289427503827                   | 2020  USA  pegas   Xport  0.1649840211516
2020  USA  pecoal  Xport  0.0515832934321573                  | 2020  USA  pecoal  Xport  0.0545400895158

The difference between the two NPi runs:

diff -y <(dumpgdx SSP2EU-NPi_2023-09-13_21.30.30/fulldata.gdx pm_IO_trade 20[0-5][0-5],.*,) <( dumpgdx SSP2EU-NPi_2023-09-14_19.08.39/fulldata.gdx pm_IO_trade 20[0-5][0-5],.*,)  | grep "|"
2000  LAM  peoil   Xport  0.4723384622313                     | 2000  LAM  peoil   Xport  0.470018383157424
2000  LAM  pegas   Xport  0.012473398035                      | 2000  LAM  pegas   Xport  0.0123428527722812
2000  LAM  pecoal  Xport  0.0404480624583                     | 2000  LAM  pecoal  Xport  0.039913965210299
2000  OAS  peoil   Xport  0.2456914436634                     | 2000  OAS  peoil   Xport  0.244484631975059
2000  OAS  pegas   Xport  0.085019564574                      | 2000  OAS  pegas   Xport  0.0841297588159855
2000  OAS  pecoal  Xport  0.0473559572157                     | 2000  OAS  pecoal  Xport  0.0467306445335058
2000  SSA  peoil   Xport  0.2687667317055                     | 2000  SSA  peoil   Xport  0.267446576520512
2000  SSA  pegas   Xport  0.00586704762                       | 2000  SSA  pegas   Xport  0.00580564372101534
2000  SSA  pecoal  Xport  0.0627762162867                     | 2000  SSA  pecoal  Xport  0.0619472864858423
2000  EUR  peoil   Xport  0.4476470458173                     | 2000  EUR  peoil   Xport  0.445448248500272
2000  EUR  pegas   Xport  0.0649559344161                     | 2000  EUR  pegas   Xport  0.0642761124862858
2000  EUR  pecoal  Xport  0.0368530617063                     | 2000  EUR  pecoal  Xport  0.0363664347174786
2000  NEU  peoil   Xport  0.2175655374438                     | 2000  NEU  peoil   Xport  0.216496877381193
2000  NEU  pegas   Xport  0.055943089587                      | 2000  NEU  pegas   Xport  0.0553575951365749
2000  NEU  pecoal  Xport  0.0005863743438                     | 2000  NEU  pecoal  Xport  0.000578631552074319
2000  MEA  peoil   Xport  1.4748738633384                     | 2000  MEA  peoil   Xport  1.46762943109214
2000  MEA  pegas   Xport  0.100203939297                      | 2000  MEA  pegas   Xport  0.0991552154813822
2000  MEA  pecoal  Xport  0.0005332600938                     | 2000  MEA  pecoal  Xport  0.000526218650248509
2000  REF  peoil   Xport  0.3334824765069                     | 2000  REF  peoil   Xport  0.331844444084994
2000  REF  pegas   Xport  0.259808582187                      | 2000  REF  pegas   Xport  0.257089453083365
2000  REF  pecoal  Xport  0.0565788494019                     | 2000  REF  pecoal  Xport  0.0558317528557609
2000  CAZ  peoil   Xport  0.1562103432897                     | 2000  CAZ  peoil   Xport  0.155443053776842
2000  CAZ  pegas   Xport  0.12207186243                       | 2000  CAZ  pegas   Xport  0.120794271247006
2000  CAZ  pecoal  Xport  0.1882728969723                     | 2000  CAZ  pecoal  Xport  0.185786843746642
2000  CHA  peoil   Xport  0.0330195481644                     | 2000  CHA  peoil   Xport  0.0328573594610639
2000  CHA  pegas   Xport  0.002664058572                      | 2000  CHA  pegas   Xport  0.00263617681714826
2000  CHA  pecoal  Xport  0.0560998470207                     | 2000  CHA  pecoal  Xport  0.0553590754710635
2000  IND  peoil   Xport  0.010888177083                      | 2000  IND  peoil   Xport  0.0108346954510287
2000  IND  pecoal  Xport  0.000772633176                      | 2000  IND  pecoal  Xport  0.000762430925807144
2000  JPN  peoil   Xport  0.005939818899                      | 2000  JPN  peoil   Xport  0.00591064310530096
2000  JPN  pecoal  Xport  0.0024792323976                     | 2000  JPN  pecoal  Xport  0.00244649532392489
2000  USA  peoil   Xport  0.0670900573707                     | 2000  USA  peoil   Xport  0.0667605177489728
2000  USA  pegas   Xport  0.007399934388                      | 2000  USA  pegas   Xport  0.00732248745845661
2000  USA  pecoal  Xport  0.0508641670635                     | 2000  USA  pecoal  Xport  0.050192530154353

See /p/tmp/oliverr/remind/compScen-exportfix-2023-09-15_11.39.46-H12-short.pdf (still running)

I thought it may fix more than that, but the only reporting variable that is affected are the PE|Production|Net|Oil and |Gas variables because in this line trade is affecting own consumption coefficients in extraction sector. These PE|Production|Net|* variables now don't complain anymore in the checks done with this script.

Type of change

• Bug fix

Checklist:

• My code follows the coding etiquette
• I performed a self-review of my own code
• I explained my changes within the PR, particularly in hard-to-understand areas
• All automated model tests pass (FAIL 0 in the output of make test)
• The changelog CHANGELOG.md has been updated correctly

[orichters]

I looked at the Trade results in the cs2, but I don't fully understand them. Should global trade not be zero? On the other hand, 0.5 EJ/yr is really small, right?

[mleimbach]

Global trade should be close to zero, but can slightly be different. In the end it is just a numerical approach of market clearing. In the validation, we always should check that this imbalance is not more than 5% of total trade in the respective traded good.