Randomness of PopulationSim outputs related to API calls

[xiex0055]

Recently, while executing MWCOG Population Synthesizer, developed based on PopulationSim v0.4.3, MWCOG staff discovered that the software consistently produced varying outputs at the disaggregate level (specifically, within the synthesized household and person files), despite identical inputs. This outcome was unexpected, considering that the entropy maximaizing and integerization methods utilized in PopulationSim should render deterministic results. In response, MWCOG staff sought assistance from their consultant, RSG. Initially, RSG staff suspected that the issue might be linked to multiprocessing (e.g., see #150). However, they subsequently realized that the multiprocessing feature was introduced in the latest version (v 0.5.1) of PopulationSim and was unavailable in the version (v0.4.3) used in the MWCOG Population Synthesizer. Aditya Gore (RSG) made the following observation:

"The PopulationSim uses linear programming tools from the ortools package for integerization. Time limits can be specified for these tools and in PopulationSim the time limit is currently set to 60 seconds. I have noticed that based on available computer resources the tool sometimes (randomness here) hit this time limit without returning a solution in which case PopulationSim turns to a different method of integerization. It is possible that you are running into this issue and each run is producing slightly different results."

MWCOG staff pose the following questions:

1. Does the randomness of PopulationSim outputs significantly impact the PopulationSim results at the aggregate level including the validation results?
2. Is there any evidence suggesting that this variability in PopulationSim outputs will NOT considerably affect regional travel demand modeling results?
3. Can this source of random be eliminated in a future version of PopulationSim?

[bettinardi]

Quickly adding that Oregon's testing and work with PopulationSim is also producing this issue (it exists for Oregon too).

[jeffreyhood]

Although I'm not yet fully apprised of the issue, my initial reaction to this information is to propose that we change the limit from a time to a number of iterations, or that we change solvers if a limit based on a number of iterations is not possible in ortools. It will be very important for statewide collaboration that the population synthesizer generate reproducible results.

[gac55]

Hello everyone - curious if you have looked into this any further?

The source of the stochasticity is unclear to me. For us, if we are doing small sample populations (~10%) this can be problematic, but for larger samples (30%+) it is far less of a problem as it begins to wash out. Understanding the variability has helped us to manage where/when we need to be careful but for many use cases it is manageable.

Has anyone been able to produce a reproducible example? A maintenance update to population sim is ongoing with a PR soon, and if it can be diagnosed with a simple fix it would be good to get in.

@gac55 @xiex0055

FWIW, I've added a config parameter that can be set in the settings.yaml to override the default hardcoded timeout of 60s.

[xiex0055]

FWIW, I've added a config parameter that can be set in the settings.yaml to override the default hardcoded timeout of 60s.

Could you point to where that parameter is, and I assume that by overriding the default timeout limit of 60s PopulationSim will be able to generate identical outputs given the same input? Also, would that overriding significantly increase PopulationSim runtime or even make the program run forever (waiting for the response of API calls)?

[xiex0055]

MWCOG staff recently conducted ActivitySim-based regional travel demand model runs with three different sets of synthetic population data that were generated using PopulationSim with the same input, with everything else being equal. We found that variations on travel demand modeling results associated with the randomness of the PopulationSim outputs were moderate (we're talking about within 1% at the regional level and within 2% at the jurisdictional level). Nevertheless, we did observe increasing variations on modeling results as we got to a finer geographic or demographic level.

Thanks @xiex0055 , FWIW PR #192 should hopefully make this randomness resolvable by allowing users to override the timeout limit for the optimizer.

[bhargavasana]

@xiex0055 have you had a chance to try #192 and reduce variation?