The PyCOMPSs AutoParallel module.
The implementation includes:
- The
@parallel()decorator for Python to run applications with PyCOMPSs - Different Translator modules to automatically generate parallel Python code
- Py2Scop Translator: Translation of Python code into OpenScop format
- Scop2PScop2Py Translator: Integration with the PLUTO tool to generate possible parallelizations of the given code. The output is written in Python (by means of a CLooG extension) with parallel annotations following an OMP fashion.
- Py2PyCOMPSs Translator: Translation of parallel annotated Python code into PyCOMPSs task definitions.
- A Code Replacer module to replace the user code by autogenerated code.
- Several example applications (
examples/folder).
COMPSs: The COMP Superscalar (COMPSs) framework is mainly compose of a programming model which aims to ease the development of applications for distributed infrastructures, such as Clusters, Grids and Clouds and a runtime system that exploits the inherent parallelism of applications at execution time. The framework is complemented by a set of tools for facilitating the development, execution monitoring and post-mortem performance analysis. It natively supports Java but has bindings for Python, C and C++.
PLUTO: PLUTO is an automatic parallelization tool based on the polyhedral model. The polyhedral model for compiler optimization provides an abstraction to perform high-level transformations such as loop-nest optimization and parallelization on affine loop nests. Pluto transforms C programs from source to source for coarse-grained parallelism and data locality simultaneously. The core transformation framework mainly works by finding affine transformations for efficient tiling. OpenMP parallel code for multicores can be automatically generated from sequential C program sections. The outer (communication-free), inner, or pipelined parallelization is achieved purely with OpenMP parallel for pragmas.
DILL: It is recommended to install the DILL Python package for better serialization
OpenSCOP: OpenScop is an open specification defining a file format and a set of data structures to represent a static control part (SCoP for short), i.e., a program part that can be represented in the polyhedral model. The goal of OpenScop is to provide a common interface to various polyhedral compilation tools in order to simplify their interaction. The OpenScop aim is to provide a stable, unified format that offers a durable guarantee that a tool can use an output or provide an input to another tool without breaking a toolchain because of some internal changes in one element of the chain. The other promise of OpenScop is the ability to assemble or replace the basic blocks of a polyhedral compilation framework at no, or at least low engineering cost.
CLooG: CLooG is a free software and library to generate code for scanning Z-polyhedra. That is, it finds a code (e.g. in C, FORTRAN...) that reaches each integral point of one or more parameterized polyhedra. CLooG has been originally written to solve the code generation problem for optimizing compilers based on the polytope model. Nevertheless, it is used now in different areas e.g. to build control automata for high-level synthesis or to find the best polynomial approximation of a function. CLooG may help in any situation where scanning polyhedra matters. While the user has full control of generated code quality, CLooG is designed to avoid control overhead and to produce a very effective code.
- Inspect Python module
- AST Python module
- Enum34 Python module
- AST Observe/Rewrite (ASTOR) Python module
- IslPy Python module
- SymPy Python module
- Uses the subprocess Python module (
scop2pscop2pytranslator) - Logging Python module
- UnitTest Python module
- To run all tests you require the Nose Python module
- To add code coverage you require coverage and/or codacy-coverage Python modules
The examples/ folder contains a folder per example application.
Each example application contains:
README.md: Description of the application and the generated task graphs.run.sh: Script to launch all the versions of the application (internally calls therun.shscript of each version of the application).- A folder per version of the application (for instance,
userparallelorautoparallel).
Each version of an example application contains:
results: Folder containing the execution resultsrun.sh: Script to launch the application locally using 4 cores*.py: Python source files of the application
Some important notes:
- All the
run.shscripts work without parameters. However, advanced users can call the scripts with extra parameters that are appended directly to theruncompsscommand. - The
autoparallelversions contain a special*_autogen.pyfile containing the automatically generated code by the COMPSs AutoParallel module. This file is only saved for clarity purposes but is re-generated every time the application is launched. - The
examples/folder contains arun.shscript to run all the available applications locally using 4 cores. It displays a result table at the end of the execution.
With debug mode enabled:
export PYTHONPATH=${git_base_dir}
cd pycompss
python nose_tests.py -s
With debug mode disabled:
export PYTHONPATH=${git_base_dir}
cd pycompss
python -O nose_tests.py
Run coverage:
./coverage_run.sh
Upload coverage:
echo "YOUR_TOKEN" > .CODACY_PROJECT_TOKEN
echo "YOUR_TOKEN" > .CODECOV_PROJECT_TOKEN
./coverage_upload.sh
This project follows the PyCodeStyle guide (formerly called pep8).
This project tolerates the following relaxations:
E501 line too long: Code lines can be up to 120 characters
You can verify the code style by running:
pycodestyle . --max-line-length=120 --statistics --count
find . -name "*.pyc" -delete
find . -name "*.pyo" -delete
All kinds of contributions are welcome. Please do not hesitate to open a new issue, submit a pull request or contact the author if necessary.
Cristián Ramón-Cortés Vilarrodona <cristian.ramoncortes(at)bsc.es> (Personal Website)
This work is supervised by:
This is part of a collaboration between the CAMUS Team at INRIA and the Workflows and Distributed Computing Team at BSC and is still under development.
Licensed under the Apache 2.0 License