Benchmarks for Alternative Implementations, Inline, and InlineIfLambda

[TheAngryByrd]

Proposed Changes

Benchmarks for Alternative Implementations, Inline, and InlineIfLambda

InlineIfLambda RFC and PR

Types of changes

What types of changes does your code introduce to MyLib.1?
Put an x in the boxes that apply

• Breaking change (fix or feature that would cause existing functionality to not work as expected) I'm not sure if this exactly a breaking change but as it touches a lot of the code, I'm just going to assume a major bump.

Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code.

• Build and tests pass locally
• I have added tests that prove my fix is effective or that my feature works (if appropriate)
• I have added necessary documentation (if appropriate)

Further comments

If this is a relatively large or complex change, kick off the discussion by explaining why you chose the solution you did and what alternatives you considered, etc...

[TheAngryByrd]

Benchmarks exploration

Types of Implementations:

• Normal
  • Reuse functions, for example, apply can be implemented in terms of bind
• Alternative
  • Don't reuse functions, do the match implementation in the body

Variations for each:

• inline
  • Which is known to give performance boosts
• InlineIfLambda
  • new perfomance technique if F# 6.0

I've chosen a few functions, specifically eitherMap and map2 as they call into other functions typically and inlining all the match statements, using inline, and the new InlineIfLambda should give some interesting results.

And as I've benchmarked and done deep analysis, I've notices the composition operator >> creates a lot of code where as a lambda will produce a lot less.

• Additional variations
  • No Composition Operator

Composition doesn't apply the Alt implementations so it leaves us with:

• EitherMap
  • Result Normal EitherMap
  • Result Normal No Composition EitherMap
  • Result Normal Inline EitherMap
  • Result Normal Inline No Composition EitherMap
  • Result Normal InlineIfLambda EitherMap
  • Result Normal InlineIfLambda NoComposition EitherMap
  • Result Alt EitherMap
  • Result Alt Inline EitherMap
  • Result Alt InlineIfLambda EitherMap
• Map2
  • Result Normal Map2
  • Result Normal No Composition Map2
  • Result Normal Inline Map2
  • Result Normal Inline No Composition Map2
  • Result Normal InlineIfLambda Map2
  • Result Normal InlineIfLambda No Composition Map2
  • Result Alt Map2
  • Result Alt Inline Map2
  • Result Alt InlineIfLambda Map2

I'm going to show the benchmarks first but then explore each of these in detail below by showing the F# Code, the decompiled code in C# and any interesting findings. I've included a SharpLab link for each so you can see the code for yourself.

Benchmark Results

EitherMap

Method	Mean	Error	StdDev	Ratio	RatioSD	Gen 0	Allocated
Result_Normal_EitherMap	19.833 ns	0.3458 ns	0.3235 ns	1.00	0.00	0.0057	48 B
Result_Normal_NoComposition_EitherMap	20.158 ns	0.1837 ns	0.1434 ns	1.01	0.02	0.0057	48 B
Result_Inlined_EitherMap	3.275 ns	0.0408 ns	0.0341 ns	0.16	0.00	-	-
Result_Inlined_NoComposition_EitherMap	3.237 ns	0.0957 ns	0.2402 ns	0.16	0.01	-	-
Result_InlinedLambda_EitherMap	3.003 ns	0.0472 ns	0.0441 ns	0.15	0.00	-	-
Result_Normal_InilnedLambda_NoComposition_EitherMap	3.449 ns	0.0953 ns	0.1904 ns	0.18	0.01	-	-
Result_Alt_EitherMap	3.301 ns	0.0908 ns	0.1637 ns	0.16	0.01	-	-
Result_Alt_Inilned_EitherMap	3.349 ns	0.0876 ns	0.1464 ns	0.17	0.01	-	-
Result_Alt_InilnedLambda_EitherMap	3.337 ns	0.0922 ns	0.1488 ns	0.17	0.01	-	-

Map2

Method	Mean	Error	StdDev	Ratio	RatioSD	Gen 0	Allocated
Result_Normal_Map2	56.549 ns	0.8351 ns	0.6974 ns	1.00	0.00	0.0172	144 B
Result_NoComposition_Map2	55.577 ns	0.6339 ns	0.5619 ns	0.98	0.02	0.0172	144 B
Result_Inlined_Map2	23.692 ns	0.2581 ns	0.2414 ns	0.42	0.01	0.0038	32 B
Result_Inlined_NoComposition_Map2	23.905 ns	0.2586 ns	0.2292 ns	0.42	0.01	0.0038	32 B
Result_InlinedLambda_Map2	11.263 ns	0.2560 ns	0.6327 ns	0.21	0.01	0.0038	32 B
Result_InlinedLambda_NoComposition_Map2	12.034 ns	0.1359 ns	0.1271 ns	0.21	0.00	0.0038	32 B
Result_Alt_Map2	6.182 ns	0.0666 ns	0.0623 ns	0.11	0.00	-	-
Result_Alt_Inlined_Map2	3.108 ns	0.0294 ns	0.0260 ns	0.06	0.00	-	-
Result_Alt_InlinedLambda_Map2	3.127 ns	0.0437 ns	0.0409 ns	0.06	0.00	-	-

Result Normal EitherMap

This is base case implementation. It doesn't try to do anything fancy, in fact it tries to re-use existing functions so the implmentations don't vary. However as we can see, this seems to be leading to some performance issues.

Sharplab link

Interesting Findings (rnem)

I do find the C# decomplilation of the Example_EitherMap intersting as you can see it has many calls and classes it has to make to get to the actual matter at hand. By contrast in later variations, this will be less convoluted.

Result Normal No Composition EitherMap

This is removing the >> (composition operator) calls.

Sharplab link

Interesting Findings (rnncem)

Nothing really different from the Normal implementation.

Result Normal Inline EitherMap

This is only adding the inline keyword to all functions associated with eitherMap.

Sharplab link

Interesting Findings (rniem)

Example_EitherMap looks a lot less complicated and the performance represents that.

Result Normal Inline No Composition EitherMap

This is adding the inline keyword to all functions associated with eitherMap and removing the >> (composition operator) calls.

Sharplab link

Interesting findings (rnincem)

While there's no noticible performance gains, there's quite a bit less C# code by comparison.

Result Normal InlineIfLambda EitherMap

This is adding the InlineIfLambda attribute to the function parameters.

Sharplab link

Interesting findings (rniilem)

No performance differences from inline and the C# code doesn't look much different either.

Result Normal InlineIfLambda No Composition EitherMap

This is adding the inline keyword to all functions associated with eitherMap, adding the InlineIfLambda attribute to the function parameters and removing the >> (composition operator) calls.

Sharplab link

Interesting findings (rniilncem)

Again no noticible perfomance differences, however the C# code is greatly reduced and very readable compared to any of the previous variations. It doesn't seem to be generating classes and other static methods.

Result Alt EitherMap

Instead of writing this in terms of the either function, this will do the match statement in the body of the function.

Sharplab link

Interesting findings (raem)

Again no noticible perfomance differences, but is making classes again.

Result Alt Inline EitherMap

Same with the previous variation except it also includes the inline keyword.

Sharplab link

Interesting findings (raiem)

Again no noticible perfomance differences, but with like the [Result Normal InlineIfLambda NoComposition EitherMap], it produces simple C# code.

Result Alt InlineIfLambda EitherMap

Same with the previous variation except it also includes the InlineIfLambda attribute.

Sharplab link

Interesting findings (raiilem)

Again no noticible perfomance differences, but with like the [Result Normal InlineIfLambda NoComposition EitherMap], it produces simple C# code.

Result Normal Map2

Sharplab link

Interesting findings (rnm2)

As expected, a lot of code and a slow benchmark

Result Normal No Composition Map2

Sharplab link

Interesting findings (rnncm2)

As expected, a lot of code and a slow benchmark

Result Normal Inline Map2

Sharplab link

Interesting findings (rnim2)

While it performed better, the code generated is even more than the non inlined versions

Result Normal Inline No Composition Map2

Sharplab link

Interesting findings (rnincm2)

Still quite a bit of code generated

Result Normal InlineIfLambda Map2

Sharplab link

Interesting findings (rniilm2)

Faster benchmarks again, Example_Map2 is pretty small in terms of code generation

Result Normal InlineIfLambda No Composition Map2

Sharplab link

Interesting findings (rniifncm2)

No change in benchmarks, nor much change in code generation

Result Alt Map2

Sharplab link

Interesting findings (ram2)

Benchmarks are improved, as the code genratred is much smaller

Result Alt Inline Map2

Sharplab link

Interesting findings (raim2)

Benchmarks are improved, as the code genratred is much smaller

Result Alt InlineIfLambda Map2

Sharplab link

Interesting findings (raiilm2)

No improvements to benchmarks or code generation

[TheAngryByrd]

Given the Alt implementation and InlineIfLambda seems to give the best performance benefits, it's time to change the implementation of these functions.

[TheAngryByrd]

After reading the docs on InlineIfLambda

An opt-in warning (/warnon:3517, off by default) can be turned on to indicate places in your code where InlineIfLambda arguments are not bound to lambda expressions at call sites. In normal situations, this warning should not be enabled. However, in certain kinds of high-performance programming, it can be useful to ensure all code is inlined and flattened.

Turning this on for the benchmarks.fsproj gives quite a few warnings. Going to keep digging at how to use this properly.

[TheAngryByrd]

let inline map ([<InlineIfLambda>] mapper : 'ok -> 'ok2) value  =
    match value with
    | Ok x -> Ok(mapper x)
    | Error e -> Error e


let inline bind ([<InlineIfLambda>] binder: 'ok -> Result<'ok2, 'err>) value  =
    match value with
    | Ok x -> binder x
    | Error e -> Error e

warning FS3517: The value 'mapper' was marked 'InlineIfLambda' but was not determined to have a lambda value. This warning is for informational purposes only
warning FS3517: The value 'binder' was marked 'InlineIfLambda' but was not determined to have a lambda value. This warning is for informational purposes only.

---

Does this mean these implementations can't take advantage of this attribute? cc @baronfel @dsyme @mrange

[mrange]

@TheAngryByrd TBH I haven't looked closely on this but my guess is that you get this error message when you pass a function value that is not a lambda. Something like this:

bind x.MyBind v

x.MyBind is a function that matches the signature but is not a lambda and thus won't be inlined.

AFAIK if you provide a lambda it shouldn't give you the warning but I haven't tried.

bind (fun x -> doSomeStuff x) v

The warning seems to be generated here:
https://github.com/dotnet/fsharp/blob/edd4c3b902bb28af15460813a823346fb4c17ad3/src/fsharp/Optimizer.fs#L2754

PS. the performance results looks promising AFAICT.

[TheAngryByrd]

[<MemoryDiagnoser>]
type MapBenchmarks () =
    [<Benchmark(Baseline = true)>]
    member this.Result_Normal_Map ()  = 
        Result.map (fun x -> x + 2 ) (Ok 1)  : Result<_,int>

    [<Benchmark>]
    member this.Result_Alt_InlinedLambda_Map ()  = 
        Result.Alt.InlinedLambda.map (fun x -> x + 2 ) (Ok 1)  : Result<_,int>


[<MemoryDiagnoser>]
type BindBenchmarks () =
    [<Benchmark(Baseline = true)>]
    member this.Result_Normal_Bind ()  = 
        Result.bind (fun x -> Ok(x + 2) ) (Ok 1) : Result<int,int>

    [<Benchmark>]
    member this.Result_Alt_InlinedLambda_Bind ()  = 
        Result.Alt.InlinedLambda.bind (fun x -> Ok(x + 2) ) (Ok 1) : Result<int,int>

This is how I'm using it currently, which is using a lambda which is why I'm a bit confused 😕

[TheAngryByrd]

Now it really seems to shine in Computation Expressions.

Method	Mean	Error	StdDev	Ratio	RatioSD	Gen 0	Allocated
Result_Normal_Bind_CE	33.60 ns	0.666 ns	0.623 ns	1.00	0.00	0.0029	24 B
Result_Alt_Inlined_Bind_CE	24.83 ns	0.523 ns	1.136 ns	0.76	0.04	-	-
Result_Alt_InlinedLambda_Bind_CE	11.74 ns	0.263 ns	0.543 ns	0.36	0.01	-	-

Normal

SharpLab link

Inline

SharpLab link

InlineIfLambda

SharpLab link

Interesting findings

The InlineIfLambda produces the cleanest code, as if you were writing the Result.bind |> Result.bind code.

[kerams]

This goes to show how big of a deal InlineIfLambda is. We get serious performance improvements while still writing clean, idiomatic F# code. Now there's even less need to resort to "imperative nonsense" :-).

Really nice work here.