hpc-graph

Hybrid parallel approach combining OpenMPI and Numba in Python to compute PageRank and Shortest Path Algorithms on Large Graphs

Specifications

Host's specifications:

• OS: GNU/Linux Ubuntu Server 24.04.3 LTS
• Architecture: x86_64
• CPU: Intel i7-7700
  • CPU(s): 8
  • Thread(s) per core: 2
  • Core(s) per socket: 4
  • Socket(s): 1
• RAM: 32 GB
• Swap space: 8 GB
• Python: 3.12.3
• GCC: 13.3.0

How to build it

Clone the repository and install the requirements:

git clone https://github.com/mastronardo/hpc-graph.git
cd hpc-graph
pip install -r requirements.txt

You can download the edge list from Network Datasets available for the "Network Science" book by Albert-László Barabási.

How to use it

To run the code, use the following command:

mpiexec -n [numprocs] python3 -m mpi4py hpc_graph.py --threads [numthreads]

Where numprocs is the number of MPI processes and numthreads is the number of threads per process.

To know the number of physical and logical cpu cores available on your host, use:

# Linux distribution
lscpu

# macOS
sysctl hw.physicalcpu hw.logicalcpu

To know how many slots has MPI available, use:

mpirun -n 1 --display-allocation hostname

If your host support hyper-threading (as the Intel i7-7700), you can use all the hardware threads by setting the flag --use-hwthread-cpus to the mpiexec command.

mpiexec -n [numprocs] --use-hwthread-cpus python3 -m mpi4py hpc_graph.py

In this case we can omit the --threads argument, because it will be set automatically to 1 by the code to not exceed the hardware specifications.