Benchmark

[DevinR528]

Add criterion bench marking for some Enforcer methods and constructor. I would imagine there are more methods and features to benchmark just let me know!

Starts to complete benchmark #5 in roadmap.

[GopherJ]

Hello we have another feature: runtime-tokio which doesn't use async-std. Is there a general solution?

[DevinR528]

I can implement a small block_on for just the benchmarks so its totally independent, this also has the benefit of possibly measuring the work being done a bit more accurately, maybe (with no spin_loop_hint type stuff the awaited task will end as soon as it is finished not the next time a Waker checks it) I doubt it would make a difference.

Basically we need to be able to run the benchmarks with tokio and with async-std correct?

[DevinR528]

Hmm it actually does seem to make some difference, but it would be more "real world" accurate to use tokio or async-std. I'll push both option so you can look it over.

crate instance of Enforcer                                                                             
                        time:   [5.5743 ms 5.6057 ms 5.6389 ms]
                        change: [+40.503% +52.834% +61.678%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 7 outliers among 100 measurements (7.00%)
  1 (1.00%) low severe
  4 (4.00%) low mild
  2 (2.00%) high mild

enforces two permissions                                                                            
                        time:   [451.13 us 454.68 us 459.50 us]
                        change: [+8.7505% +14.485% +19.593%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 9 outliers among 100 measurements (9.00%)
  6 (6.00%) high mild
  3 (3.00%) high severe

adds permission for user MemoryAdapter                                                                            
                        time:   [469.03 ns 469.66 ns 470.22 ns]
                        change: [+13.310% +13.705% +14.103%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 6 outliers among 100 measurements (6.00%)
  1 (1.00%) low severe
  4 (4.00%) low mild
  1 (1.00%) high mild

crate instance of FileAdapter                                                                                     
                        time:   [0.0000 ps 0.0000 ps 0.0000 ps]
                        change: [+1.4584% +112.87% +339.56%] (p = 0.06 > 0.05)
                        No change in performance detected.
Found 12 outliers among 100 measurements (12.00%)
  4 (4.00%) high mild
  8 (8.00%) high severe

Benchmarking crate instance of DefaultModel: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 13.6s or reduce sample count to 40.
crate instance of DefaultModel                                                                             
                        time:   [2.6901 ms 2.6904 ms 2.6906 ms]
                        change: [-0.6876% -0.3564% +0.0011%] (p = 0.04 < 0.05)
                        Change within noise threshold.
Found 7 outliers among 100 measurements (7.00%)
  1 (1.00%) low severe
  6 (6.00%) high severe

this is run with async-std against the simple block_on so +percentage = slower

[codecov]

Codecov Report

Merging #68 into master will not change coverage by %.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master      #68   +/-   ##
=======================================
  Coverage   80.46%   80.46%           
=======================================
  Files          18       18           
  Lines        2580     2580           
=======================================
  Hits         2076     2076           
  Misses        504      504           

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e9c2260...af3df94. Read the comment docs.

[GopherJ]

LGTM! and thanks for your great work.