Benchmark results are likely dominated by overheads instead of actual computation

[yebai]

Most existing models are fairly lightweight computationally; thus, the benchmarking numbers are slightly misleading since they are mostly dominated by various overheads in DynamicPPL rather than actual computation.

See, e.g., chalk-lab/Mooncake.jl#571 (comment) for an example of Mooncake's performance often being similar to Enzyme and better than compiled ReverseDiff.

[penelopeysm]

The main question here is whether it's expected that for larger models the results of the benchmarks will change substantially. Even for the n100 and n500 models Mooncake is still slower than both Enzyme and RD (although those might not be very representative since it's the same calculation being done over and over again).

If yes, then the solution would be to add some larger models, separate the table into different groups (e.g. real-life models vs. coverage DynamicPPL features), and add some explanatory text.

If not, then I wouldn't consider this to be a problem, because the table doesn't claim to show the raw performance of a given AD backend on generic functions (I never set out to do that here - that is a bigger issue than this repo and it's not our job to provide such a comparison). It only shows the performance on DynamicPPL models and any overhead that DPPL introduces is part of that metric.

If the answer to that isn't known, then more models should be added to find that out.

[yebai]

Can you add models in https://arxiv.org/abs/2002.02702 as a separate table/page?

[yebai]

chalk-lab/Mooncake.jl#552 is one of the causes of Mooncake's overheads.