Add pre-commit for continuous integration

Author: Zhijian Liu

.github/workflows/formatter.yml

@@ -0,0 +1,17 @@
+name: pre-commit
+
+on:
+  pull_request:
+  push:
+    branches: [master]
+
+jobs:
+  pre-commit:
+    runs-on: ubuntu-latest
+    steps:
+    - run: |
+        sudo apt-get update
+        sudo apt-get install -y --no-install-recommends clang-format
+    - uses: actions/checkout@v2
+    - uses: actions/setup-python@v2
+    - uses: pre-commit/action@v2.0.3

.github/workflows/pre-commit.yml

@@ -1,3 +1,3 @@
 .vscode/
 build/
-*.pyc
+*.pyc

.gitignore

@@ -0,0 +1,63 @@
+repos:
+  - repo: https://github.com/pre-commit/pre-commit-hooks
+    rev: v4.0.1
+    hooks:
+      - id: trailing-whitespace
+        name: (Common) Remove trailing whitespaces
+      - id: mixed-line-ending
+        name: (Common) Fix mixed line ending
+        args: ['--fix=lf']
+      - id: end-of-file-fixer
+        name: (Common) Remove extra EOF newlines
+      - id: check-merge-conflict
+        name: (Common) Check for merge conflicts
+      - id: requirements-txt-fixer
+        name: (Common) Sort "requirements.txt"
+      - id: fix-encoding-pragma
+        name: (Python) Remove encoding pragmas
+        args: ['--remove']
+      - id: double-quote-string-fixer
+        name: (Python) Fix double-quoted strings
+      - id: debug-statements
+        name: (Python) Check for debugger imports
+      - id: check-json
+        name: (JSON) Check syntax
+      - id: check-yaml
+        name: (YAML) Check syntax
+      - id: check-toml
+        name: (TOML) Check syntax
+  - repo: https://github.com/asottile/pyupgrade
+    rev: v2.19.4
+    hooks:
+      - id: pyupgrade
+        name: (Python) Update syntax for newer versions
+        args: ['--py36-plus']
+  - repo: https://github.com/google/yapf
+    rev: v0.31.0
+    hooks:
+      - id: yapf
+        name: (Python) Format with yapf
+  - repo: https://github.com/pycqa/isort
+    rev: 5.8.0
+    hooks:
+      - id: isort
+        name: (Python) Sort imports with isort
+  - repo: https://github.com/pycqa/flake8
+    rev: 3.9.2
+    hooks:
+      - id: flake8
+        name: (Python) Check with flake8
+        additional_dependencies: [flake8-bugbear, flake8-comprehensions, flake8-docstrings, flake8-executable, flake8-quotes]
+  - repo: https://github.com/pre-commit/mirrors-mypy
+    rev: v0.902
+    hooks:
+      - id: mypy
+        name: (Python) Check with mypy
+        additional_dependencies: [tokenize-rt]
+  - repo: local
+    hooks:
+      - id: clang-format
+        name: (C/C++/CUDA) Format with clang-format
+        entry: clang-format -style=google -i
+        language: system
+        files: \.(h\+\+|h|hh|hxx|hpp|cuh|c|cc|cpp|cu|c\+\+|cxx|tpp|txx)$

.pre-commit-config.yaml

@@ -46,4 +46,4 @@ SOFTWARE.
 
 Please cite "4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural
 Networks", CVPR'19 (https://arxiv.org/abs/1904.08755) if you use any part
-of the code.
+of the code.

LICENSE

@@ -8,7 +8,7 @@
 
 ## Overview
 
-We release `torchsparse`, a high-performance computing library for efficient 3D sparse convolution. This library aims at accelerating sparse computation in 3D, in particular the Sparse Convolution operation. 
+We release `torchsparse`, a high-performance computing library for efficient 3D sparse convolution. This library aims at accelerating sparse computation in 3D, in particular the Sparse Convolution operation.
 
 <img src="https://hanlab.mit.edu/projects/spvnas/figures/sparseconv_illustration.gif" width="1080">
 
@@ -52,7 +52,7 @@ inds, labels, inverse_map = sparse_quantize(pc, feat, labels, return_index=True,
 
 where `pc`, `feat`, `labels` corresponds to point cloud (coordinates, should be integer), feature and ground-truth. The `inds` denotes unique indices in the point cloud coordinates, and `inverse_map` denotes the unique index each point is corresponding to. The `inverse map` is used to restore full point cloud prediction from downsampled prediction.
 
-To combine a list of `SparseTensor`s to a batch, you may want to use the `torchsparse.utils.sparse_collate_fn` function. 
+To combine a list of `SparseTensor`s to a batch, you may want to use the `torchsparse.utils.sparse_collate_fn` function.
 
 Detailed results are given in [SemanticKITTI dataset preprocessing code](https://github.com/mit-han-lab/e3d/blob/master/spvnas/core/datasets/semantic_kitti.py) in our [SPVNAS](https://github.com/mit-han-lab/e3d) project.
 
@@ -99,7 +99,7 @@ In this example, `sphash` is the function converting integer coordinates to hash
 
 ### Dummy Training Example
 
-We here provides an entire training example with dummy input [here](examples/example.py). In this example, we cover 
+We here provides an entire training example with dummy input [here](examples/example.py). In this example, we cover
 
 - How we start from point cloud data and convert it to SparseTensor format;
 - How we can implement SparseTensor batching;
@@ -109,7 +109,7 @@ You are also welcomed to check out our [SPVNAS](https://github.com/mit-han-lab/e
 
 ### Mixed Precision (float16) Support
 
-Mixed precision training is supported via `torch.cuda.amp.autocast` and `torch.cuda.amp.GradScaler`. Enabling mixed precision training can speed up training and reduce GPU memory usage. By wrapping your training code in a `torch.cuda.amp.autocast` block, feature tensors will automatically be converted to float16 if possible. See [here](examples/example.py) for a complete example. 
+Mixed precision training is supported via `torch.cuda.amp.autocast` and `torch.cuda.amp.GradScaler`. Enabling mixed precision training can speed up training and reduce GPU memory usage. By wrapping your training code in a `torch.cuda.amp.autocast` block, feature tensors will automatically be converted to float16 if possible. See [here](examples/example.py) for a complete example.
 
 ## Speed Comparison Between torchsparse and MinkowskiEngine
 

README.md

@@ -1,11 +1,13 @@
+import argparse
+
 import numpy as np
 import torch
 import torch.nn as nn
-import torchsparse
+
 import torchsparse.nn as spnn
 from torchsparse import SparseTensor
-from torchsparse.utils import sparse_collate_fn, sparse_quantize
-import argparse
+from torchsparse.utils.collate import sparse_collate_fn
+from torchsparse.utils.quantize import sparse_quantize
 
 
 def generate_random_point_cloud(size=100000, voxel_size=0.2):
@@ -35,7 +37,7 @@ def generate_batched_random_point_clouds(size=100000,
                                          voxel_size=0.2,
                                          batch_size=2):
     batch = []
-    for i in range(batch_size):
+    for _ in range(batch_size):
         batch.append(generate_random_point_cloud(size, voxel_size))
     return sparse_collate_fn(batch)
 
@@ -45,7 +47,7 @@ def dummy_train(device, mixed=False):
         spnn.Conv3d(4, 32, kernel_size=3, stride=1), spnn.BatchNorm(32),
         spnn.ReLU(True), spnn.Conv3d(32, 64, kernel_size=2, stride=2),
         spnn.BatchNorm(64), spnn.ReLU(True),
-        spnn.Conv3d(64, 64, kernel_size=2, stride=2, transpose=True),
+        spnn.Conv3d(64, 64, kernel_size=2, stride=2, transposed=True),
         spnn.BatchNorm(64), spnn.ReLU(True),
         spnn.Conv3d(64, 32, kernel_size=3, stride=1), spnn.BatchNorm(32),
         spnn.ReLU(True), spnn.Conv3d(32, 10, kernel_size=1)).to(device)
@@ -71,12 +73,12 @@ def dummy_train(device, mixed=False):
 
 if __name__ == '__main__':
     parser = argparse.ArgumentParser()
-    parser.add_argument("--mixed", action="store_true")
+    parser.add_argument('--mixed', action='store_true')
     args = parser.parse_args()
 
     # set seeds for reproducibility
     np.random.seed(2021)
     torch.manual_seed(2021)
 
     device = 'cuda:0' if torch.cuda.is_available() else 'cpu'
-    dummy_train(device, args.mixed)
+    dummy_train(device, args.mixed)

examples/example.py

@@ -3,10 +3,14 @@
 import numpy as np
 import torch
 import torch.autograd.profiler as profiler
+import torch.cuda
 import torch.nn as nn
+import torch.optim
+
 import torchsparse.nn as spnn
 from torchsparse import SparseTensor
-from torchsparse.utils import sparse_collate_fn, sparse_quantize
+from torchsparse.utils.collate import sparse_collate_fn
+from torchsparse.utils.quantize import sparse_quantize
 
 
 def generate_random_point_cloud(size=100000, voxel_size=0.2):
@@ -36,7 +40,7 @@ def generate_batched_random_point_clouds(size=100000,
                                          voxel_size=0.2,
                                          batch_size=2):
     batch = []
-    for i in range(batch_size):
+    for _ in range(batch_size):
         batch.append(generate_random_point_cloud(size, voxel_size))
     return sparse_collate_fn(batch)
 
@@ -47,19 +51,19 @@ def dummy_train_3x3(device):
         spnn.Conv3d(32, 64, kernel_size=3, stride=1),
         spnn.Conv3d(64, 128, kernel_size=3, stride=1),
         spnn.Conv3d(128, 256, kernel_size=3, stride=1),
-        spnn.Conv3d(256, 128, kernel_size=3, stride=1, transpose=True),
-        spnn.Conv3d(128, 64, kernel_size=3, stride=1, transpose=True),
-        spnn.Conv3d(64, 32, kernel_size=3, stride=1, transpose=True),
-        spnn.Conv3d(32, 10, kernel_size=3, stride=1, transpose=True),
+        spnn.Conv3d(256, 128, kernel_size=3, stride=1, transposed=True),
+        spnn.Conv3d(128, 64, kernel_size=3, stride=1, transposed=True),
+        spnn.Conv3d(64, 32, kernel_size=3, stride=1, transposed=True),
+        spnn.Conv3d(32, 10, kernel_size=3, stride=1, transposed=True),
     ).to(device)
     optimizer = torch.optim.Adam(model.parameters(), lr=0.001)
     criterion = nn.CrossEntropyLoss().to(device)
 
     print('Starting dummy_train_3x3...')
     time = datetime.now()
     with profiler.profile(profile_memory=True, use_cuda=True) as prof:
-        with profiler.record_function("model_inference"):
-            for i in range(10):
+        with profiler.record_function('model_inference'):
+            for _ in range(10):
                 feed_dict = generate_batched_random_point_clouds()
                 inputs = feed_dict['lidar'].to(device)
                 targets = feed_dict['targets'].F.to(device).long()
@@ -69,8 +73,8 @@ def dummy_train_3x3(device):
                 loss.backward()
                 optimizer.step()
                 # print('[step %d] loss = %f.'%(i, loss.item()))
-    print(prof.key_averages().table(sort_by="cuda_time_total", row_limit=10))
-    prof.export_chrome_trace("trace_dummy_3x3.json")
+    print(prof.key_averages().table(sort_by='cuda_time_total', row_limit=10))
+    prof.export_chrome_trace('trace_dummy_3x3.json')
 
     time = datetime.now() - time
     print('Finished dummy_train_3x3 in ', time)
@@ -82,19 +86,19 @@ def dummy_train_3x1(device):
         spnn.Conv3d(32, 64, kernel_size=(1, 3, 3), stride=1),
         spnn.Conv3d(64, 128, kernel_size=(3, 1, 3), stride=1),
         spnn.Conv3d(128, 256, kernel_size=(1, 3, 3), stride=1),
-        spnn.Conv3d(256, 128, kernel_size=(3, 1, 3), stride=1, transpose=True),
-        spnn.Conv3d(128, 64, kernel_size=(1, 3, 3), stride=1, transpose=True),
-        spnn.Conv3d(64, 32, kernel_size=(3, 1, 3), stride=1, transpose=True),
-        spnn.Conv3d(32, 10, kernel_size=(1, 3, 3), stride=1, transpose=True),
+        spnn.Conv3d(256, 128, kernel_size=(3, 1, 3), stride=1, transposed=True),
+        spnn.Conv3d(128, 64, kernel_size=(1, 3, 3), stride=1, transposed=True),
+        spnn.Conv3d(64, 32, kernel_size=(3, 1, 3), stride=1, transposed=True),
+        spnn.Conv3d(32, 10, kernel_size=(1, 3, 3), stride=1, transposed=True),
     ).to(device)
     optimizer = torch.optim.Adam(model.parameters(), lr=0.001)
     criterion = nn.CrossEntropyLoss().to(device)
 
     print('Starting dummy_train_3x1 ...')
     time = datetime.now()
     with profiler.profile(profile_memory=True, use_cuda=True) as prof:
-        with profiler.record_function("model_inference"):
-            for i in range(10):
+        with profiler.record_function('model_inference'):
+            for _ in range(10):
                 feed_dict = generate_batched_random_point_clouds()
                 inputs = feed_dict['lidar'].to(device)
                 targets = feed_dict['targets'].F.to(device).long()
@@ -104,8 +108,8 @@ def dummy_train_3x1(device):
                 loss.backward()
                 optimizer.step()
                 # print('[step %d] loss = %f.'%(i, loss.item()))
-    print(prof.key_averages().table(sort_by="cuda_time_total", row_limit=10))
-    prof.export_chrome_trace("trace_dummy_3x1.json")
+    print(prof.key_averages().table(sort_by='cuda_time_total', row_limit=10))
+    prof.export_chrome_trace('trace_dummy_3x1.json')
 
     time = datetime.now() - time
     print('Finished dummy_train_3x1 in ', time)

examples/performance.py

@@ -0,0 +1,15 @@
+[yapf]
+based_on_style = google
+spaces_around_power_operator = true
+split_before_arithmetic_operator = true
+split_before_logical_operator = true
+split_before_bitwise_operator = true
+
+[isort]
+known_first_party = torchsparse, torchsparse.backend
+
+[flake8]
+select = B, C, E, F, P, T4, W, B9
+ignore = E501, E722, W503
+per-file-ignores =
+    __init__.py: F401, F403