List of applications: apps.json. The currently available apps are earthquake and flood. Configurations for each contained within apps.json.
Each configuration has several elements:
name
: Identifier to link the configuration JSON with the client-requested HTML file (e.g. seismic.html, seisme.html, flood.html, inondation.html). This allows the configuration and language to be set.mapView
: Defaults for where the map should be centered and the zoom level.baseLayers
: Provides a list of base layers that the application can use.welcome
: Contains the HTML code for the welcome modulemodals
: Provides the contents for each of the modals.form
: Provides content for the new simulation form (note: check if this is still being used; I suspect not).simComplete
: Contains the HTML code for the simulation complete moduletools
: Gives tools such asmap-click
(detailed later) and the associated actions.
Here are the five containers that currently comprise ER2:
Container | Server | Port | Description |
---|---|---|---|
er2_front_end (this repo) | Flask | xxxx | Front-end for ER2 homepage, ER2 Flood, and ER2 Earthquake (serving HTML, JS, static files, forms, etc.). Routes are / (ER2 homepage), /earthquake , /seisme , /flood , /inondation , /form/<hazard> , and /toggleLanguage . |
mapserver | 81 | wms service for ER2 base layers and result layers. | |
flood_calculations | Flask | xxxy | Routes are /initiate , /query , /tiffValue , and /status/<task_id> . The route /initiate calls the function calculate_damages_task ; the route /status/<task_id> retrieves the status as the simulation progresses. Celery is used for task queuing. |
earthquake_calculations | This container is managed by | ||
earthquake_gfm | This container is managed by |
This repo is for the er2_front_end container (the other containers are listed to give greater context).
The Dockerfile:
FROM python:3.7-alpine
# Don't want to use default user; create our own
RUN adduser -D er2
WORKDIR /home/er2
# Put Python dependencies in container
COPY requirements.txt requirements.txt
# Create virtual environment and install all the requirements in it
RUN python -m venv venv
RUN venv/bin/pip install -r requirements.txt
# gunicorn web server; this could also be placed in requirements.txt
# mysql needed too
RUN venv/bin/pip install gunicorn pymysql
# Install the application in the container
COPY app app
COPY migrations migrations
COPY er2.py config.py boot.sh ./
# Ensures that boot.sh is set as an executable file
RUN chmod +x boot.sh
ENV FLASK_APP er2.py
# Sets the owner of all files in /home/er2 as the new er2 user
RUN chown -R er2:er2 ./
# Make the er2 user the default for subsequent instructions
USER er2
# The flood front end container will run on port <port>
EXPOSE <port>
# The default command when the container is started (in a seperate script for simplicity)
ENTRYPOINT ["./boot.sh"]
The boot.sh
file:
#!/bin/sh
# this script is used to boot a Docker container
source venv/bin/activate
while true; do
flask db upgrade
if [[ "$?" == "0" ]]; then
break
fi
echo Deploy command failed, retrying in 5 secs...
sleep 5
done
flask translate compile
exec gunicorn -b :<docker-port> --access-logfile - --error-logfile - er2:app
To build a container image: $ docker build -t er2:latest .
You can run the container with the docker run
command, which takes a number of arguments:
docker run --name er2 -d -p 8000:<docker-port> --rm -e SECRET_KEY=<secret-key> er2:latest
The --name
option provides a name for the new container. The -d
option tells Docker to run the container in the background (so the command prompt is not blocked). The -p
option maps container ports to host ports. The first port is the port on the host computer, and the one on the right is inside the container. Here we are exposing port in the container on port 8000 in the host, so you will access the application on 8000, even though internally the container is using . The -rm
option will delete the container once it is terminated. The -e
option is for setting run-time environment variables. The last argument is the container image name and tag used to use for the container. After you run the above command, you can access the application at http://localhost:8000
The output of docker run
is the container ID. If you want to see what containers are running, you can use docker ps
.
To update strings that need to be translated:
pybabel extract -F babel.cfg -k _l -o messages.pot .
Then apply the updates by:
pybabel update -i messages.pot -d app/translations
Then edit the messages.po
file. Finally, apply the changes:
pybabel compile -d app/translations
This compile command adds a mesages.mo file next to messages.po in each language repo. The .mo file is the file that Flask-Babel uses to load translations.
SQLALchemy is used as the object relational mapper (ORM). The database for user login credentials is SQLite. To access database, enter Python console and enter the following commands:
from app import create_app
app = create_app()
app.app_context().push()
from app.models import User
# For example, to query all users
User.query.all()
To add user:
u = User(first_name='fn', last_name='ln', affiliation='affiliation', username='username',
email='email@email.com')
u.set_password('password goes here')
from app import db
db.session.add(u)
db.session.commit()
This repo contains a folder migrations
for the database migration scripts. This is generatd my Flask-Migrate/Alembic. To create this migration repository, if it doesn't exist, run flask db init
.
The configuration file for earthquake and flood includes a list of base layers. These base layers are loaded by the application on start up. The JavaScript cycles through the list and makes a layer out of each (there is all the neccesary info for making an OpenLayers layer). Here is an example of a layer configuration:
{
"legend_name": "Gatineau",
"id": "gat_blocks",
"opacity": "1",
"zIndex": 5,
"makeVis": true,
"service": "http://localhost:81/cgi-bin/mapserv",
"ratio": "1",
"type": "image",
"params": {
"LAYERS": "gat_blocks",
"MAP": "/map/er2/flood.map"
},
"serverType": "mapserver",
"crossOrigin": "anonymous",
"additional_info": {
"legendURL": "http://localhost:81/cgi-bin/mapserv?map=/map/er2/flood.map&SERVICE=WMS&VERSION=1.3.0&REQUEST=getlegendgraphic&FORMAT=image/png&sld_version=1.1.0&layer=gat_blocks",
"getFeatureInfoOnClick": false
}
}
For the earthquake module, the base layers are (1) the OpenStreetMap (OSM) layer; (2) the census tract polygons; (3) historical earthquakes; and (4) seismic regions. For the flood module, the base layers are (1) the OSM layer, (2) the census block polygons, (3) the HAND model, and (4) the boundary for the HAND model.
Note: For the moment, the OSM base layer is added directly via the JavaScript (i.e. hard coded and not included in the base layer list above). This is not a good practice and will be changed in the future.
In the configuration, under tools
, there is a handler for map-click
.
For the earthquake module:
"tools": [
{
"type": "map-click",
"name": "Generate quake",
"action": "http://localhost:<local-port>/form/seismic?_="
}
]
For the flood module:
"tools": [
{
"type": "map-click",
"name": "Generate flood",
"action": "http://localhost:<local-port>/form/flood?_="
}
]
This tells the client what to do on a map click event. Note that for flood the action is not correct — the TIFF value should be fetched first, rather than the form.
With a map click, the x,y
coordinates are recorded (for both the earthquake and flood modules). For the flood module, the HAND value of the clicked location is then fetched, and then the form is displayed. for the earthquake module, the form is displayed.
The HAND value for the clicked location is first fetched. The x,y
coordinates are sent to the back-end service, i.e. http://localhost:<local-port>/tiffValue?x=${x}&y=${y}
.
This route calls a function (getTiffValue
) that determines the value of the raster for the x,y
coordinate. It executes the GDAL command gdallocationinfo.exe
(GDAL comes installed with QGIS). Note: In the future, we may eliminate this route and instead use Sherbrooke's service (likely more convienent and logical).
If the x,y
coordinates are outside the HAND geoTIFF boundary, the HAND value cannot be obtained. The user is notified of the error and instructed to select a different location. If the coordinates are valid and a HAND value is returned, the hazard specification form is requested (i.e. GET
request with x
, y
, and srs
as parameters). For example:
forms/flood?_=1554236346005&x=-75.67502975&y=45.46843792&xRounded=-75.6750&yRounded=45.4684&srs=EPSG%3A4326
The resultant form, which opens within a modal, simply shows the clicked location and prompts the user for the flood depth:
The form is submitted (i.e. POST
request) to http://<server-name>:<local-port>/initiate
. This URL route is detailed later in this document.
With the earthquake module, a map click results in the form being requested (i.e. GET
request with x
, y
, and srs
as parameters). For example:
<forms/eq?_=1554236849731&x=-72.50976563&y=49.17272589&xRounded=-72.5098&yRounded=49.1727&srs=EPSG%3A4326>
The form has two options:
The form is submitted via a POST
request to http://<server-name>:8080/gfm/rs/ratt/process/eq
.
For asynchronous task queuing, ER2-Flood uses Celery. Celery has three core components:
- The Celery client issues background jobs.
- The Celery workers are the processes that run the background jobs.
- The message broker is the client that communicates with the the workers through a message queue (in this case, Redis). This is needed to store and send task states.
The Flask application initializes the Celery client by creating an object of class Celery and passing the application name and the connection URL for the message broker:
from flask import Flask, request, jsonify, url_for
from celery import Celery
app = Flask(__name__)
# Configure location of Redis database
app.config['CELERY_BROKER_URL'] = 'redis://localhost:<local-port>/0'
# To store the state and return values of tasks in Redis
app.config['CELERY_RESULT_BACKEND'] = 'redis://localhost:<local-port>/0'
# Initializing the Celery client
celery = Celery(app.name, broker=app.config['CELERY_BROKER_URL'])
celery.conf.update(app.config)
Functions that run as background tasks are decorated with celery.task
. The bind=True
argument is also added so that Celery sends a self argument to the function (neccesary for recording status updates). For example:
@celery.task(bind=True)
def calculate_damages_task(self, waterLevel, x, y):
# Long running task here; code omitted
return "Simulation complete!"
To kick off a task (Celery worker process), the client issues a POST
request to /initiate
(as was described earlier). Here is the /initiate
route:
@app.route("/initiate", methods=["POST"])
def initiate():
postedData = request.form.to_dict()
waterLevel = float(postedData["waterLevel"])
x = float(postedData["x"])
y = float(postedData["y"])
# Request the execution of this background task
# Return value of apply_async() is an object that represents the task
task = calculate_damages_task.apply_async(args=[waterLevel, x, y])
task_status_url = "http://<server-name>:<host-port>" + url_for(
"taskstatus", task_id=task.id
)
response = {
"type": "flood-task-status",
"taskid": task.id,
"task_status_url": task_status_url,
"progress": "0",
"status": "not-started",
}
return jsonify(response)
The task_status_url
is included in the return and tells the client where to obtain status information. This takes the format of a URL (e.g. /status/<task_id>
where task.id
is a dynamic component) that points to another Flask route called taskstatus
. The taskstatus
route (shown below) reports the status of the background tasks. It returns a JSON that includes the task state and all the values set in update_state()
.
@app.route("/status/<task_id>", methods=["GET"])
def taskstatus(task_id):
task = calculate_damages_task.AsyncResult(task_id)
if task.state == "PENDING":
# job did not start yet
response = {
"state": task.state,
"actions": None,
"current": 0,
"total": 1,
"progress": 0,
"status": "not-started",
}
elif task.state != "FAILURE":
response = {
"state": task.state,
"actions": task.info.get("actions", 0),
"current": task.info.get("current", 0),
"total": task.info.get("total", 0),
"progress": int(
(task.info.get("current", 0) / task.info.get("total", 0)) * 100
),
"status": task.info.get("status", 0),
}
return jsonify(response)
Note that Celery receives task updates through self.update_state()
, within the calculate_damages_task()
function:
# Within the calculate_damages_task() function
self.update_state(
state="PROGRESS",
meta={
"actions": actions,
"current": ticker,
"total": progressOutOf,
"status": "ongoing",
"progress": progress,
},
)
Above, the state is PROGRESS
. The client uses current
and total
to display a progress bar. The actions
contains information on each of the layers to be loaded (for brevity the full action list is not shown here).
For further information on Celery and Flask: https://blog.miguelgrinberg.com/post/using-celery-with-flask
The earthquake module uses a different task queuing service. The documentation has not yet been completed.
However, here is a sample response after the form is submitted:
{
"type": "epicenter-task-status",
"taskid": "<guid>",
"task_status_url": "http://<server-name>:8080/gfm/rs/tasks/<guid>",
"progress": 0,
"status": "not-started",
"actions": [
{
"type": "marker",
"id": "<guid>",
"action": "add",
"srs": "4326",
"link": "null",
"geometry": {
"type": "Point",
"coordinates": [-72.72949219, 49.13140841]
}
}
]
}
These responses are continually sent (upon a GET
request from the client to the task_status_url
) until the simulation completes (i.e. progress equals 100).
The database connection is . There are two databases: er2wps_results (earthquake) and flood_results_test (flood).
Note that the inventory is not held in this database. For flood, the inventory is held memory as csv files.
The flood database is called flood_results_test
(note: this database is no longer the test; it should be renamed). ER2 Flood uses two tables contained in this database, flood_simulation
and flood_sim_result
.
Tables/Views | Description |
---|---|
flood_result_view_2 | Flood simulation results |
flood_simulation | Record of simulations (sim_id, sim_started, sim_percent_completed, etc.) |
flood_sim_result | Flood simulation results with geometry (view used by Mapserver) |
In the table flood_simulation
, a record is made for each simulation (i.e. simulation ID, status, start time, completion time, percent completed, water depth, latitude, and longitude):
This record is updated as the simulation proceeds.
In the table flood_sim_result
the results for each affected block are recorded. Presently there are fields for the building count, building exposure, content exposure, total exposure, structural damage, content damage, total damage, buildings affected, population, and affected population:
Database SQL commands are done from Python (psycopg2
library).
The database also has a table for the census blocks base layer (gatineaublocks
). The geometry field is queried by Mapserver.
The earthquake database is er2wps_results.
Tables/Views | Description |
---|---|
hz_tract | Census tracts (2006) |
secan_r2 | Earthquake regions |
xyearthquakes_cleaned_2 | Historical earthquakes |
job_result | Earthquake simulation results |
er2wps_view | Earthquake simulation results with geometry (view used by Mapserver) |
An table overviewing the earthquake database was presented above ("ER2 architecture overview").
The Postgres database is er2wps_results
. Tables/views include:
-
hz_tract: the geometry for the census tracts
-
secan_r2: the polygons for seismic regions
-
xyearthquakes_cleaned_2: the historical earthquake points
-
er2wps_view: this is queried by MapServer to display result layers, including injuries (2AM, 2PM, 5PM), fatalities (2AM, 2PM, 5PM), ssfa, s1fv, pga, soil class, and economic loss
The application has the URL to check the status (recall that this was returned by the /initiate
or /eq
function). Every 2.0 seconds a GET
request is made to this URL (/status/<task_id>
), and the server responds with a JSON in the format shown below. The current
, progress
, and total
fields are used to show a progress bar. The actions
are used to show layers.
For the flood module:
{
"actions": [
{
"source": [
{
"id": "<unique_id>_total_dmg",
"legend_name": "Total Damage",
"legend_url": "http://<server-name>:81/cgi-bin/mapserv?map=/map/er2/flood.map&SERVICE=WMS&VERSION=1.3.0&REQUEST=getlegendgraphic&FORMAT=image/png&sld_version=1.1.0&layer=total_dmg",
"name": "total_dmg",
"query_info_url": "<server-name>:<host-port>/query?id=<unique_id>",
"serverType": "mapserver",
"service": "http://<server-name>:81/cgi-bin/mapserv?map=/map/er2/flood.map",
"simId": "<unique_id>",
"styles": "",
"type": "wms-layer"
},
{
"id": "<unique_id>_bldgs_affected",
"legend_name": "Buildings Affected",
"legend_url": "http://<server-name>:81/cgi-bin/mapserv?map=/map/er2/flood.map&SERVICE=WMS&VERSION=1.3.0&REQUEST=getlegendgraphic&FORMAT=image/png&sld_version=1.1.0&layer=bldgs_affected",
"name": "bldgs_affected",
"query_info_url": "http://<server-name>:<host-port>/query?id=<unique_id>",
"serverType": "mapserver",
"service": "http://<server-name>:81/cgi-bin/mapserv?map=/map/er2/flood.map",
"simId": "<unique_id>",
"styles": "",
"type": "wms-layer"
},
{
"id": "<unique_id>_population",
"legend_name": "Population",
"legend_url": "http://<server-name>:81/cgi-bin/mapserv?map=/map/er2/flood.map&SERVICE=WMS&VERSION=1.3.0&REQUEST=getlegendgraphic&FORMAT=image/png&sld_version=1.1.0&layer=population",
"name": "population",
"query_info_url": "http://<server-name>:<host-port>/query?id=<unique_id>",
"serverType": "mapserver",
"service": "http://<server-name>:81/cgi-bin/mapserv?map=/map/er2/flood.map",
"simId": "<unique_id>",
"styles": "",
"type": "wms-layer"
},
{
"id": "8962183579_affected_population",
"legend_name": "Affected Population",
"legend_url": "http://<server-name>:81/cgi-bin/mapserv?map=/map/er2/flood.map&SERVICE=WMS&VERSION=1.3.0&REQUEST=getlegendgraphic&FORMAT=image/png&sld_version=1.1.0&layer=affected_population",
"name": "affected_population",
"query_info_url": "http://<server-name>:<host-port>/query?id=8962183579",
"serverType": "mapserver",
"service": "http://<server-name>:81/cgi-bin/mapserv?map=/map/er2/flood.map",
"simId": "8962183579",
"styles": "",
"type": "wms-layer"
}
],
"type": "load-layer"
}
],
"current": 8,
"progress": 15,
"state": "PROGRESS",
"status": "ongoing",
"total": 59
}
Similarly, for the earthquake module:
{
"type": "epicenter-task-status",
"taskid": "<guid>",
"task_status_url": "<server-name>:8080/gfm/rs/tasks/<guid>",
"progress": 88,
"status": "ongoing",
"actions": [
{
"type": "load-layer",
"id": "<guid>_load_layer",
"source": [
{
"type": "wms-layer",
"service": "<server-name>:8080/geoserver/trit_test/wms",
"label": "Total economic loss",
"key": "RATT-econ-loss",
"name": "<guid>_src",
"layers": "trit_test:er2wps_view_copy",
"styles": "RATT-econ-loss",
"params": "cql_filter:job_id='job_<job-id>'",
"serverType": "geoserver",
"info": {
"type": "gfm-world-info",
"url": "<server-name>:8080/gfm/rs/view/json/ratt/rsim_gettractbyxy?job_id=job_<job-id>",
"mime_type": "application/json"
}
},
{
"type": "wms-layer",
"service": "<server-name>:8080/geoserver/trit_test/wms",
"label": "2 AM injuries",
"key": "injuries_2am",
"name": "<guid>_layer_2",
"layers": "trit_test:er2wps_view_copy",
"styles": "RATT-injuries_2am",
"params": "cql_filter:job_id='job_<job-id>'",
"serverType": "geoserver",
"info": {
"type": "gfm-world-info",
"url": "<server-name>:8080/gfm/rs/view/json/ratt/rsim_gettractbyxy?job_id=job_<job-id>",
"mime_type": "application/json"
}
},
{
"type": "wms-layer",
"service": "<server-name>:8080/geoserver/trit_test/wms",
"label": "2 PM injuries",
"key": "injuries_2pm",
"name": "<guid>_layer_3",
"layers": "trit_test:er2wps_view_copy",
"styles": "RATT-injuries_2pm",
"params": "cql_filter:job_id='job_<job-id>'",
"serverType": "geoserver",
"info": {
"type": "gfm-world-info",
"url": "<server-name>:8080/gfm/rs/view/json/ratt/rsim_gettractbyxy?job_id=job_<job-id>",
"mime_type": "application/json"
}
},
{
"type": "wms-layer",
"service": "<server-name>:8080/geoserver/trit_test/wms",
"label": "5 PM injuries",
"key": "injuries_5pm",
"name": "<guid>_layer_4",
"layers": "trit_test:er2wps_view_copy",
"styles": "RATT-injuries_5pm",
"params": "cql_filter:job_id='job_<job-id>'",
"serverType": "geoserver",
"info": {
"type": "gfm-world-info",
"url": "<server-name>:8080/gfm/rs/view/json/ratt/rsim_gettractbyxy?job_id=job_<job-id>",
"mime_type": "application/json"
}
},
{
"type": "wms-layer",
"service": "<server-name>:8080/geoserver/trit_test/wms",
"label": "2 AM fatalities",
"key": "fatal_2am",
"name": "<guid>_layer_9",
"layers": "trit_test:er2wps_view_copy",
"styles": "RATT-fatal_2am",
"params": "cql_filter:job_id='job_<job-id>'",
"serverType": "geoserver",
"info": {
"type": "gfm-world-info",
"url": "<server-name>:8080/gfm/rs/view/json/ratt/rsim_gettractbyxy?job_id=job_<job-id>",
"mime_type": "application/json"
}
},
{
"type": "wms-layer",
"service": "<server-name>:8080/geoserver/trit_test/wms",
"label": "2 PM fatalities",
"key": "fatal_2pm",
"name": "<guid>_layer_10",
"layers": "trit_test:er2wps_view_copy",
"styles": "RATT-fatal_2pm",
"params": "cql_filter:job_id='job_<job-id>'",
"serverType": "geoserver",
"info": {
"type": "gfm-world-info",
"url": "<server-name>:8080/gfm/rs/view/json/ratt/rsim_gettractbyxy?job_id=job_<job-id>",
"mime_type": "application/json"
}
},
{
"type": "wms-layer",
"service": "<server-name>:8080/geoserver/trit_test/wms",
"label": "5 PM fatalities",
"key": "fatal_5pm",
"name": "<guid>_layer_11",
"layers": "trit_test:er2wps_view_copy",
"styles": "RATT-fatal_5pm",
"params": "cql_filter:job_id='job_<job-id>'",
"serverType": "geoserver",
"info": {
"type": "gfm-world-info",
"url": "<server-name>:8080/gfm/rs/view/json/ratt/rsim_gettractbyxy?job_id=job_<job-id>",
"mime_type": "application/json"
}
},
{
"type": "wms-layer",
"service": "<server-name>:8080/geoserver/trit_test/wms",
"label": "PGA",
"key": "RATT-pga",
"name": "<guid>_layer_5",
"layers": "trit_test:er2wps_view_copy",
"styles": "RATT-pga",
"params": "cql_filter:job_id='job_<job-id>'",
"serverType": "geoserver",
"info": {
"type": "gfm-world-info",
"url": "<server-name>:8080/gfm/rs/view/json/ratt/rsim_gettractbyxy?job_id=job_<job-id>",
"mime_type": "application/json"
}
},
{
"type": "wms-layer",
"service": "<server-name>:8080/geoserver/trit_test/wms",
"label": "Sa @ 0.3 s",
"key": "RATT-ssfa",
"name": "<guid>_layer_6",
"layers": "trit_test:er2wps_view_copy",
"styles": "RATT-ssfa",
"params": "cql_filter:job_id='job_<job-id>'",
"serverType": "geoserver",
"info": {
"type": "gfm-world-info",
"url": "<server-name>:8080/gfm/rs/view/json/ratt/rsim_gettractbyxy?job_id=job_<job-id>",
"mime_type": "application/json"
}
},
{
"type": "wms-layer",
"service": "<server-name>:8080/geoserver/trit_test/wms",
"label": "Sa @ 1.0 s",
"key": "RATT-s1fv",
"name": "<guid>_layer_7",
"layers": "trit_test:er2wps_view_copy",
"styles": "RATT-s1fv",
"params": "cql_filter:job_id='job_<job-id>'",
"serverType": "geoserver",
"info": {
"type": "gfm-world-info",
"url": "<server-name>:8080/gfm/rs/view/json/ratt/rsim_gettractbyxy?job_id=job_<job-id>",
"mime_type": "application/json"
}
},
{
"type": "wms-layer",
"service": "<server-name>:8080/geoserver/trit_test/wms",
"label": "Soil type",
"key": "RATT-site",
"name": "<guid>_layer_8",
"layers": "trit_test:er2wps_view_copy",
"styles": "RATT-soil_char",
"params": "cql_filter:job_id='job_<job-id>'",
"serverType": "geoserver",
"info": {
"type": "gfm-world-info",
"url": "<server-name>:8080/gfm/rs/view/json/ratt/rsim_gettractbyxy?job_id=job_<job-id>",
"mime_type": "application/json"
}
}
]
},
{
"type": "marker",
"id": "<guid>",
"action": "add",
"srs": "4326",
"link": "null",
"geometry": { "type": "Point", "coordinates": [-71.433, 48.524] }
}
]
}
See CONTRIBUTING.md
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
The Canada wordmark and related graphics associated with this distribution are protected under trademark law and copyright law. No permission is granted to use them outside the parameters of the Government of Canada's corporate identity program. For more information, see Federal identity requirements.