Subway Station Hazard Detection, Goethe University Frankfurt (Spring 2020)

General Information

Instructors:

• Prof. Dr. Visvanathan Ramesh, email: V.Ramesh@em.uni-frankfurt.de
• Dr. Michael Rammensee, email: M.Rammensee@em.uni-frankfurt.de

Institutions:

• Goethe University
• AISEL - AI Systems Engineering Lab

Project team (A-Z):

• Pascal Fischer
• Felix Hoffman
• Edis Kurtanovic
• Martin Ludwig
• Alen Smajic

Publications

• ResearchGate

Tools

• Python 3
• PyTorch Framework
• Unity3D
• C#
• Blender
• OpenCV

Project Description

Recently there was a nationwide scandal about an incident at Frankfurt Central Station in which a boy was pushed onto the railroad tracks in front of an arriving train and lost his life due to its injuries. However, this incident is not the only one of its kind because such accidents occur from time to time at train stations. At the moment there are no security systems to prevent such accidents, and cameras only exist to provide evidence or clarification after an incident has already happened. In this project, we developed a first prototype of a security system which uses the surveillance cameras at subway stations in combination with the latest deep learning computer vision methods and integrates them into an end-to-end system, which can recognize dangerous situations and initiate measures to prevent further consequences. Furthermore, we present a 3D subway station simulation, developed in Unity3D, which generates entire train station environments and scenes using an advanced algorithm trough a real data-based distribution of persons. This simulation is then used to generate training data for the deep learning model.

Subway Station Simulation

For our simulation we developed 10 different types of subway stations, covering the most common station architectures. The number of platforms varies between 1 and 2, while the number of tracks lies between 1 and 4, for each station. Every station type was manually textured in 5 different variations to get a total of 50 unique station environments.

Furthermore, we include a variety of different human models as well as station objects like benches, snack machines, stairs, rubbish bins etc. Using our Script-UI you can further expand the amount of different station objects and station types. Once you press on the Unity play button, the algorithm starts to generate the station environments by randomly generating and placing the human models and station objects along the subway station. Once the scenario is generated, the algorithm takes a screenshot and saves the image to a predefined folder within the project folder. Since we are training a semantic segmentation algorithm we also need to generate the ground truth labels. To do so, our algorithm replaces all station objects and the station itself with white textured versions of those objects. The human models are replaced with green, yellow and red versions of the human models based on their location within the subway station. If they are staying on the railroads, they are painted in red. If they are staying in front of the security line, they are painted in yellow. In all other cases, the human models are replaced with green ones. Finally, our algorithm takes another screenshot and stores the image as the ground truth label in a separate folder within the project folder.

On the right side you can see our Script-UI which is used to controll the simulation. It contains a camera object which is used to take the screenshots. Upon activating the first checkbox "Use Distribution" the simulation produces a more realistic scenario where the persons are distributed along the station in a more realistic way (less probability to stay on the railtracks). The second checkbox "Create Samples" is used to create a random scene from the simulation and freezes it (this is mostly used for testing purposes once we add new objects to the simulation). The option "Number Of Stations" defines how many station architecture types are used by the simulation (in this case 10 since we implemented 10 different station architectures). The option "Number Of Scenes" defines how many different scenes are generated on the same station environment (station architecture with unique texture environment plus station objects). The "Number of Types" option is used to define the amount of unique texture environments for each station architecture.

Semantic Segmentation using SegNet

For detecting a hazard on the subway station we are using semantic segmentation to classify each pixel of an image to one of the following classes:

• white - background
• black - security line
• green - character in save area
• yellow - character near the dangerous area
• red - character in the dangerous area

Training

To train the SegNet there are 2 scripts available:

• SUBWAY_SEGMENTATION.py
• Subwaystation_Segmentation.ipynb

For both you only need a folder with the input images and another one with the target images. To start the training you only should execute:

python3 SUBWAY_SEGMENTATION.py

with the following parameters

• --input path to img input data directory
• --target path to img target data directory
• --content path where the train/validation tensors, model_weights, losses, validation will be saved, default="/"
• --train_tensor_size number of images per training_tensor (should be True: train_tensor_size % batch_size == 0)
• --val_tensor_size help='number of images per training_tensor (should be True: train_tensor_size % batch_size == 0)
• --num_train_tensors help='number of train tensors (should be True: train_tensor_size * num_train_tensors + val_tensor_size == |images|)
• --model_weights path where your model weights will be loaded, if not defined new weights will initialized
• --epochs number of training epochs, default=50
• --batch_size batch size for training, default=8
• --learn_rate learning rate for training, default=0.0001
• --momentum momentum for stochastic gradient descent, default=0.9
• --save_cycle save model, loss, validation every save_cycle epochs, default=5
• --weight_decay weight_decay for stochastic gradient descent, default= 4e5

example execution with 41.000 input/target images:

python3 test_parse.py --input=data/training --target=data/target --content=data/output --train_tensor_size=2000 --val_tensor_size=1000 --num_train_tensors=20  model_weights=model.pt

configuration from example execution:

• input_path=data/training
• target_path=data/target
• content_path=data/output
• batch_size=2000
• train_tensor_size=20
• val_tensor_size=1000
• num_train_tensors=20
• model_weights=model.pt
• load_model=True
• learn_rate=0.0001
• momentum=0.9
• weight_decay=400000.0
• total_epochs=50
• save_cycle=5

for Subwaystation_Segmentation.ipynb its the equivalent for google-colab. You can set all the parameters in the configuration cell.

check it out:

Results