Metabolic Pathway Visualization

• Overview
• Installation
  • Dependencies
  • Database Setup
  • Server Setup
  • Server Update
• A Note On Usage

Overview

This project implements a web app for metabolic pathway visualization. The backend is written in Python using Flask and the frontend is written in HTML/CSS and JavaScript.

Backend

There backend comprises several components.

Flask

The primary component is the Flask web server which handles URL routing for webpages and all other requests from the frontend. The Flask server is implemented in the metapaths.py file. This code is responsible for serving the pages that the user visits as well as managing the search tasks.

Celery

When a user submits a pathway search, the Flask server utilizes Celery to manage that search task. Celery is a distributed task queue and allows the main server process to continue handling requests while the search is executed asynchronously by Celery workers.

MySQL Databases

The backend is also supported by two MySQL databases: metadb and hubdb. These databases can be recreated by sourcing the provided database dumps in the resources directory (see in-depth instructions in the Installation section below).

The metadb database contains a single table, KEGGCompoundNames which provides a mapping of KEGG IDs to compound names. This information is requested by the frontend and used to populate fields and label nodes in the visualization.

The hubdb database contains a table for each pair of hub compounds for which there are pathways. For example, each row in the C00022_C00024 table describes a path from C00022 to C00024. The information in this database is used to populate the hub visualization when a user clicks to expand a hub link.

Frontend

The HTML for webpages resides in the templates directory and is served by the Flask server. The frontend makes extensive use of Bootstrap for the UI components and Select2 for the searchable drop-down select boxes. The visualization is created using a force-directed graph layout from D3.

Installation

• Instructions are applicable for Debian/Ubuntu and Mac OSX.

• Python version 2.7.* is assumed.

• $ indicates command should be executed from the shell

• mysql> indicates command should be executed from the MySQL server REPL

Dependencies

Python Libraries

Flask: $ sudo pip install Flask

Celery: $ sudo pip install celery

MySQLdb: $ sudo pip install mysql-python

Other

MySQL Server: https://dev.mysql.com/downloads/mysql/

RabbitMQ: Debian/Ubuntu, Mac OSX

Clone

$ git clone https://github.com/KavrakiLab/metapaths.git

Database Setup

1. Complete the initial post-installation setup of MySQL Server. The default user is root and the password is meta. If you configure with a different username and password, you must update the DB_USER and DB_PASSWD fields in metapaths.py. Start and login to the MySQL server with:

$ mysql -u root -p

2. Create metadb database

mysql> CREATE DATABASE metadb;
mysql> USE metadb;
mysql> SOURCE ./metapaths/resources/metadb_2017-04-30.sql;

3. Create hubdb database

mysql> CREATE DATABASE hubdb;
mysql> USE hubdb;
mysql> SOURCE ./metapaths/resources/hubdb_2017-04-30.sql;

4. Logout

mysql> quit;

Server Setup

Once all the dependencies are available and the databases have been setup, we can run the main Flask server.

If using Ubuntu or Debian, the RabbitMQ server will begin running right after installation, so no further action is needed. On Mac OSX, you will need to run rabbitmq-server to start up the broker.

Next, we need to intialize Celery.

$ cd metapaths
$ celery worker -A metapaths.celery --loglevel=info

This creates a celery work for the metapaths application. The log level can be configured as desired. You can run the Celery worker in its own shell and use another shell to run the Flask server, or the worker can be run in the background and its output and logging can be redirected into a log file like this:

$ cd metapaths
$ celery worker -A metapaths.celery --loglevel=info &> celery_worker.log &

Lastly, we can start the Flask server:

$ cd metapaths
$ export FLASK_APP=metapaths.py
$ flask run

The Flask server will initialize and the webapp will become available at localhost:5000

Server Update

If we make changes to the server's javascript or static .css or .html code, we can update the server by entering the following:

$ sudo /etc/init.d/apache2 reload

If we make changes to the python scripts, we can update the server using the following process:

$ killall ../bin/python2.7
$ ../bin/celery multi start worker -A metapaths.celery  --loglevel=info --pidfile=../run/%n.pid --logfile=../log/%n%I.log
export FLASK_APP=metapaths.py
$ nohup flask run &
$ ../bin/celery multi start worker  --loglevel=info --pidfile=../run/%n.pid --logfile=../log/%n%I.log
$ sudo /etc/init.d/apache2 reload

The first line kills all existing celery workers. Then the next two lines recreates a celery worker for metapaths and restarts the Flask server. The last line restarts apache.

Deploying to a non-local server

The whole metapaths web app can be deployed to the cloud using apache and mod_wsgi. Complete instructions can be found here.

This guide can be used to setup apache and mod_wsgi, but the other installation instructions regarding dependencies, database setup, and celery will also need to be followed.

A Note On Usage

Celery and RabbitMQ are necessary for the search execution functionality of the webapp. However, it is possible to just run the Flask web app and still use all of the visualization features.

This can be done in two ways:

1. By uploading properly formatted JSON using the upload feature on the webapp

   For examples of the JSON accepted by this feature, see the resources directory. This is the same JSON format as is given when pathways are exported from the web app.

2. By placing a text file of pathways in the searches/output directory and clicking "View more pathways" on the web app.

   The format of these text files is simple; each line represents a pathway and contains a comma delimited list of KEGG compound and RPair IDs. For a sample of this format, inspect any of the files in the searches/examples directory.

   Note that this is not exactly the same format as that output by the AtomMetaNet algorithms. However, the path_convert.py tool (located in the searches directory) can be used to convert certain supported formats into this format. For example, to convert LPAT:

   python path_convert.py lpat lpat_output.txt
   

   To convert Hub output:

   python path_convert.py hub hub_output.txt
   

   Note that path_convert.py overwrites the original input file with the converted format.