Reproducible Benchmarks

[avik-pal]

Fixes #13

TODOs:

• Use deterministic point cloud generators
• Use non-inplace versions of operators
• Update Kaolin scripts

[DhairyaLGandhi]

bump

[nirmal-suthar]

I am also changing the format of PointCloud to 3xNxB, so this benchmark need to be updated. I will make necessary changes to the benchmark once I finish this change.

[nirmal-suthar]

@avik-pal I have added kaolin equivalent script for benchmark, can you please review it.

[avik-pal]

LGTM. Can you run the benchmarks and report the results here before it goes in?

[nirmal-suthar]

(Kaolin) Results of NormalizePointCloud on gpu is looking a bit odd

DEVICE: CPU
Running benchmarks for npoints = 4096
ScalePointCloud: 0.061511993408203125 ms
RotatePointCloud: 0.049591064453125 ms
RealignPointCloud: 8.382320404052734 ms
NormalizePointCloud: 0.48279762268066406 ms
Compose: 0.6380081176757812 ms

Running benchmarks for npoints = 16384
ScalePointCloud: 7.851362228393555 ms
RotatePointCloud: 7.779836654663086 ms
RealignPointCloud: 34.340858459472656 ms
NormalizePointCloud: 23.39315414428711 ms
Compose: 70.2211856842041 ms

Running benchmarks for npoints = 65536
ScalePointCloud: 7.91621208190918 ms
RotatePointCloud: 7.879734039306641 ms
RealignPointCloud: 34.55471992492676 ms
NormalizePointCloud: 23.876667022705078 ms
Compose: 70.770263671875 ms

Running benchmarks for npoints = 262144
ScalePointCloud: 8.008003234863281 ms
RotatePointCloud: 8.074760437011719 ms
RealignPointCloud: 39.08681869506836 ms
NormalizePointCloud: 26.52740478515625 ms
Compose: 75.27351379394531 ms

Running benchmarks for npoints = 1048576
ScalePointCloud: 9.147882461547852 ms
RotatePointCloud: 9.363174438476562 ms
RealignPointCloud: 74.33509826660156 ms
NormalizePointCloud: 48.63238334655762 ms
Compose: 108.71624946594238 ms

CUDA is on. Running GPU Benchmarks
DEVICE: GPU
Running benchmarks for npoints = 4096
ScalePointCloud: 0.19551999866962433 ms
RotatePointCloud: 0.10787200182676315 ms
RealignPointCloud: 0.9790719747543334 ms
NormalizePointCloud: 0.23318399488925934 ms
Compose: 1.2248640060424805 ms

Running benchmarks for npoints = 16384
ScalePointCloud: 0.1963520050048828 ms
RotatePointCloud: 0.1074879989027977 ms
RealignPointCloud: 3.4454400539398193 ms
NormalizePointCloud: 0.2341119945049286 ms
Compose: 3.7479360103607178 ms

Running benchmarks for npoints = 65536
ScalePointCloud: 0.19500799477100372 ms
RotatePointCloud: 0.12303999811410905 ms
RealignPointCloud: 13.480992317199707 ms
NormalizePointCloud: 0.25571200251579285 ms
Compose: 13.843168258666992 ms

Running benchmarks for npoints = 262144
ScalePointCloud: 0.19657599925994873 ms
RotatePointCloud: 0.30796799063682556 ms
RealignPointCloud: 80.64806365966797 ms
NormalizePointCloud: 0.25577598810195923 ms
Compose: 81.30508422851562 ms

Running benchmarks for npoints = 1048576
ScalePointCloud: 0.2795200049877167 ms
RotatePointCloud: 1.0577919483184814 ms
RealignPointCloud: 324.3732604980469 ms
NormalizePointCloud: 0.3883199989795685 ms
Compose: 326.091796875 ms

[DhairyaLGandhi]

How are you running the benchmarks?

[nirmal-suthar]

How are you running the benchmarks?

on cyclops

[nirmal-suthar]

(Flux3D) Updated

DEVICE: CPU
Running benchmarks for npoints = 4096
ScalePointCloud: 0.009655 ms
RotatePointCloud: 0.075988 ms
RealignPointCloud: 0.279625 ms
NormalizePointCloud: 0.203925 ms
Compose: 0.563456 ms

Running benchmarks for npoints = 16384
ScalePointCloud: 0.045578999999999995 ms
RotatePointCloud: 0.296556 ms
RealignPointCloud: 1.066775 ms
NormalizePointCloud: 0.7840309999999999 ms
Compose: 2.164345 ms

Running benchmarks for npoints = 65536
ScalePointCloud: 0.191927 ms
RotatePointCloud: 0.287846 ms
RealignPointCloud: 4.208021 ms
NormalizePointCloud: 3.08079 ms
Compose: 7.725478 ms

Running benchmarks for npoints = 262144
ScalePointCloud: 0.774146 ms
RotatePointCloud: 1.201146 ms
RealignPointCloud: 16.922203 ms
NormalizePointCloud: 12.431453999999999 ms
Compose: 30.959553999999997 ms

Running benchmarks for npoints = 1048576
ScalePointCloud: 3.4260509999999997 ms
RotatePointCloud: 4.014949 ms
RealignPointCloud: 66.776906 ms
NormalizePointCloud: 48.911978999999995 ms
Compose: 121.588326 ms

CUDA is on. Running GPU Benchmarks
DEVICE: GPU
Running benchmarks for npoints = 4096
ScalePointCloud: 0.049512999999999995 ms
RotatePointCloud: 0.047611999999999995 ms
RealignPointCloud: 0.716013 ms
NormalizePointCloud: 1.3064639999999998 ms
Compose: 2.177975 ms

Running benchmarks for npoints = 16384
ScalePointCloud: 0.052364999999999995 ms
RotatePointCloud: 0.07320499999999999 ms
RealignPointCloud: 0.759052 ms
NormalizePointCloud: 1.413719 ms
Compose: 2.300122 ms

Running benchmarks for npoints = 65536
ScalePointCloud: 0.062423 ms
RotatePointCloud: 0.13674899999999998 ms
RealignPointCloud: 0.7757299999999999 ms
NormalizePointCloud: 1.453902 ms
Compose: 2.316599 ms

Running benchmarks for npoints = 262144
ScalePointCloud: 0.152287 ms
RotatePointCloud: 0.393957 ms
RealignPointCloud: 0.811319 ms
NormalizePointCloud: 1.470068 ms
Compose: 2.291968 ms

Running benchmarks for npoints = 1048576
ScalePointCloud: 0.44273599999999996 ms
RotatePointCloud: 1.3051949999999999 ms
RealignPointCloud: 1.1824139999999999 ms
NormalizePointCloud: 1.757831 ms
Compose: 4.192527999999999 ms

[DhairyaLGandhi]

Cyclops can be a bit noisy, but more than that, use timeit to get your times.

[DhairyaLGandhi]

Also please add graphs for these numbers to the readme.

[nirmal-suthar]

These are the final graphs for benchmarks. I think using horizontal one would be better.

[avik-pal]

The horizontal plot is definitely the one we should go with

[nirmal-suthar]

I am thinking to add TriMesh benchmarks and graph in this PR before merging. Will rebase this once TriMesh PR is merged.

[avik-pal]

Maybe let's merge this PR before the TriMesh. We might want to do performance regressions in the benchmark PR for Meshes.

Also, don't forget to add the plots in the README

[nirmal-suthar]

Sure

[avik-pal]

@nirmal-suthar You missed adding the plot to README

[nirmal-suthar]

I have updated readme on master, missed it here.