Skip to content

Latest commit

 

History

History
executable file
·
62 lines (48 loc) · 5.07 KB

Super-Computing.md

File metadata and controls

executable file
·
62 lines (48 loc) · 5.07 KB
Raw

HPC, Distributed Computing, Cloud computing, Cluster computing, Grid computing, Parallel computing, etc..

Computer Performance

  • Boxfish :: A platform for visualizing performance data across the domains the data was collected, (e.g. communication groups) to another (e.g. hardware processes).
  • PeachPy :: A Python framework for writing high-performance assembly kernels - an efficient Assembly Code-generator in Higher-level Python.

CONCURRENCY

Cloud-Cluster

  • arrayfire-python :: A high performance library for parallel computing with an easy-to-use API.
  • clusterlib :: Tools to manage jobs on supercomputer. Documentation
  • Curio :: The coroutine concurrency library.
  • lustre_kvm_quickstart :: Support scripts for the Lustre KVM. Quick Start Guide at : http://wiki.lustre.org/KVM_Quick_Start_Guide
  • Pathos :: A framework for parallel graph management and execution in heterogeneous computing. It primarily provides the communication mechanisms for configuring and launching parallel computations across heterogenous resources. Pathos provides configurable launchers for parallel and distributed computing, where each launcher contains the syntactic logic to configure and launch jobs in an execution environment.
  • Pyina :: a MPI-based parallel mapper and launcher.
  • Ufora :: A compiled, automatically parallel subset of python for data science and numerical computing.

AWS

  • AmazonEC2scripts :: Useful scripts to setup Amazon EC2 Cloud Compute servers for Neuroimaging.
  • Anaconda-ec2 :: Anaconda plugin for StarCluster.
  • Boto :: A Python package that provides interfaces to Amazon Web Services. The source code on github.
  • Cloudman :: Easily create a compute cluster on top of a cloud computing infrastructure.
  • kappa :: Kappa is a command line tool that makes it easier to deploy, update, and test functions for AWS Lambda.
  • StarCluster is an open source cluster-computing toolkit for Amazon's Elastic Compute Cloud (EC2). Source code on Github

APACHE

  • SparklingPandas :: Make it easy to use the distributed computing power of PySpark to scale your data anlysis with Pandas.
  • Kazoo :: A high-level Python library that makes it easier to use Apache Zookeeper. Documentation.

Hadoop

  • Hue :: A Web application for interacting with Apache Hadoop that supports a file and job browser, Hive, Pig, Impala, Spark, Oozie, HBase, Solr, Sqoop2, ZooKeeper and more.
Resources

HPC

  • Dask :: Task scheduling and blocked algorithms for parallel processing. Source code and Documentation.
  • Luigi :: A Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.
  • Pyston : An open-source Python implementation using JIT techniques, built using LLVM for performance speed. Their blog introducing Pyston
  • PySGE : Low-level Python interface to Sun Grid Engine.
  • Pythran : A Static Compilation of Parallel Scientific Kernels a.k.a. Python/Numpy compiler for the mass. Source Code on Github.
Resources