2012 December Design Model Architecture

About this document

This roadmap and architecture document presented from "model developer" point of view, which imply C++ development process, user aspects of OpenM++ are deliberately excluded. Please refer to OpenM++ user guide pages for additional details.

OpenM++ model use cases

OpenM++ by design is portable and scalable environment which allow researchers to run same model on single Windows PC and on Linux (or Windows) HPC cluster by simply re-compiling model C++ code for target platform. For example, model developer can use Visual Studio on his own Windows PC to write, test and debug the model and later send model .cpp code to other researcher who can build and run that model on Linux HPC cluster with hundreds CPUs.

There are four main groups of openM++ model users:

• developer: using C++ IDE with openM++ installed to develop and run models mostly on their local PC
• researcher: uses openM++ models created by developer executable to run simulation on local workstation and/or on HPC cluster
• institutional user: member of research organization with advanced IT infrastructure who mostly running openM++ models in resource-shared environment (i.e. over the web)
• public user: member of the general public using simplified interface over the web.

Those user groups do have distinctive hardware / software environments and different requirements to model architecture:

• developer:
  • mostly local Windows or Linux PC with GUI
  • run the model hundred times to debug it
  • have full admin privileges on his local machine
  • eventually need to pack model executable and data files and send it to researcher
• researcher:
  • HPC cluster (large or small) or local Windows, Linux without GUI
  • run the model multiple times and collect the results
  • run the model 100's or 1000's of times for Probabilitistic Sensitivity Analysis or for model estimation.
  • do not have admin privileges, especially on cluster
  • often need to pack model data files to publish it, move from local PC to HPC cluster or share with other researchers
• institutional user:
  • uses web UI to run the model in cloud, on HPC cluster or other powerful server environment
  • have absolutely no access to actual server environment
  • at any time can use IT department to deploy openM++ models in cloud, create modeling web-sites, manage model database on SQL server, etc.
• public user:
  • runs a version of a model via the web written and compiled in openM++ with a limited set of parameters and limited set of output screens, possibly in parallel with hundreds of other general public users.
  • very limited if any capacity at all to save results between sessions.

It is typical for openM++ users to not have advanced IT management skill as they are highly regarded professionals in their own area of interest. It may also not always possible for openM++ user to install additional software in their environment (i.e. in public HPC cluster). From that point easiest way of model deployment and model data export-import can be done through simple file operations (file copy). It is obviously not suitable for institutional users, however they can: (a) rely on dedicated IT department resources if necessary and (b) do have installed and supported web-servers, SQL databases servers and other resources where openM++ cloud components can be deployed.

Based on those use cases openM++ model architecture assumes following preferences:

• model, input parameters and output results available as set of files
• user may not want to (or can’t install) database client-server software to store model data

Note: To simplify description of model architecture below it is done from developer or researcher user point of view and web cloud aspects are deliberately excluded.

OpenM++ model run cycle

Because openM++ models can scale from single PC to HPC cluster model execution (model run-cycle) depends on environment.

Simple (single PC) case (italic indicates optional):

• start of model executable (model.exe)
• read model settings from database (read execution scenario)
• read model input data from database
• run modeling loop:
  • execute user model code
  • report on model progress if required
• do model results aggregation if required
• write results into database output tables
• finally report execution statistics and exit

If model runs in cluster environment then openM++ can transparently create multiple copies of model executable process and distribute it on cluster nodes.

Model run-cycle on cluster (italic indicates optional):

• start of master model executable (model.exe)
• read model settings from database (read execution scenario)
• detect run-time environment
• spawn model.exe processes on computational nodes
• read model input data from database
• distribute input data between all computational nodes
• run modeling loop:
  • execute user model code
  • report on model progress if required
  • collect model tracking information to debug the model
• wait until all modeling completed on all computational nodes
• collect model results from each node
• do results aggregation if required
• write results into database output tables
• finally report execution statistics and exit

Note: It is important to understand the diagram on that page represent schematic picture and real openM++ code may be significantly more complex. For example, report modeling progress call exchangeProgress() may not actually do anything but place a data in the buffer and separate thread would do actual master-slave communication and progress report.

OpenM++ modeling library

The modeling library provides core functionality for the model run-cycle as it is described above. It contains main() entry point, it does agent creation / destruction, event queue management, on-the-fly cross-tabulation, and pre- and post-simulation processing.

It uses OpenM++ data and execute libraries to organize model execution (especially in cluster environment), read model input parameters, save model tracks and aggregate cross-tabulation results:

• for each input parameter model library by known data type, shape and other necessary information (memory address if required) to instantiate class object and populate it with values by calling data library
• for each output table result model library call data library to save results in model data storage (model database)

OpenM++ model data storage (model database)

OpenM++ data storage should provide an ability to store model parameters and output results. It consist of model data storage (model database), data library and, optionally, can use execute library to organize communication between computational nodes.

It can be implemented in following ways:

• option 0. flat files: directly read-write into flat text (XML, CSV, etc.) files
• option a. flat files + buffering (or MPI-IO): use memory buffering (or MPI-IO) to organize large size chunks and reduce data exchange in cluster environment
• option b. client-server database: use MySQL or other open source SQL server database
• option c. file-based (embedded) SQL database: use file-based database (i.e. SQLite) inside of master process and write custom code to emulate client-server for computational nodes

Evaluating those options from point of view openM++ use cases described above:

Option 0: direct write to flat files may not be realistic approach in cluster environment because:

• computational nodes most likely don’t have locale file system
• global shared file system may have very high or prohibitive cost for small write operations. For example, if 100 model executables from 100 computational nodes want write to 100 bytes it may be, in worst case, 100 times slower than if master node writes 100*100 bytes. Of course, MPI-IO can solve that problem.

Option a: flat files + buffering (or MPI-IO)

• pros:
  • most human readable format
  • no additional tools required to create or modify model data, it can be done by any text editor
  • minimal development efforts
• cons:
  • real model data input typically bigger than user can type-in and maintain without additional tools
  • to analyze the data in any other software (i.e. Excel, R, SAS) custom data converter(s) must be developed

Option b: client-server

• pros:
  • relatively easy to implement
  • good performance is almost guaranteed
  • hundreds tools to read, compare and manipulate the data
• cons:
  • require to install and administer SQL server database which many openM++ users, such as model developers and independent researchers may have no right to do or may not want to do

Option c: file-based database (i.e. SQLite)

• pros:
  • hundreds tools to read and manipulate the data (i.e. Firefox SQLite manager add-on)
  • relatively easy to transfer to any database or exchange the data between researchers
• cons:
  • development time to create client-server code for cluster environment much higher than any other options
  • it is less convenient as flat text files

OpenM++ data storage roadmap:

OpenM++ data storage can be implemented in following order of priorities:

• (pri1) inside of single embedded (file-based) SQL database
• (pri2) as above plus extra database for model tracking
• (pri3) model parameters and metadata inside of file-based SQL database and output results as .csv files
• (pri3) inside of SQL server database chosen by model developer (i.e. MSSQL, Oracle, etc.)

OpenM++ data library

Data library(s) is a C++ library to support model data read/write operations and hide low-level implementation details to simplify model code and modeling library. It is important to understand there is no "one size fit all solution" and openM++ must provide multiple versions of data library for different target model storage. For example, for model developer SQLite data library may be most convenient, however when openM++ installed as part of web solution then MySQL data library suites more.

Following priority order of data libraries implementation planned:

• (pri1) SQLite as embedded (file-based) database
• (pri2) generic ODBC tested with MySQL (MariaDB), PostgreSQL, MS SQL, Oracle and IBM DB2
• (pri3) flat text files version of data library (using MPI-IO)
• (pri3) MySQL (MariaDB) native client (non-ODBC)
• (pri3) PostgreSQL native client (non-ODBC)

List above is not final and can be changed anytime. Many other options also considered for development of specialized data library version. For example, libmysqld, Firebird, MS Access reviewed as potential candidates for embedded (file-based) database. Also MPI-IO, HDF5, NetCDF considered as foundation for flat text files data library version. And in the future releases it is very much possible to have native client (not ODBC-based) version of data library for MS SQL, Oracle and IBM DB2.

Keep in mind data library is part of the model run-time and not be ideal choice for other purpose. Most easy way to integrate openM++ with existing products is to use SQL loaders or output convertors. It allows to import or export data from openM++ data storage into other well-known SQL servers, i.e. from SQLite into MS SQL or dump it into flat text files (i.e. CSV, XML).