DLPX-78745 Apply estat iscsi approach to stbtrace script

[brad-lewis]

Recently the estat iscsi script was overhauled. The stbtrace script suffers from some of the same issues that the estat script had, most notably missing write events and reporting incorrect throughput. This commit rewrites the probe functions in the iscsi stbtrace script. It uses the same probe functions as the estat version. They have different aggregate_data functions.

For testing the output was compared to the estat output to verify it's correctness:

$ sudo stbtrace iscsi 
{"t":"1639163735", "op":"read", "count":"4114", "avgLatency":"3892728", "throughput":"269090816", "latency":"{200000,7},{300000,20},{400000,35},{500000,49},{600000,90},{700000,121},{800000,165},{900000,156},{1000000,158},{2000000,1299},{3000000,643},{4000000,303},{5000000,165},{6000000,120},{7000000,107},{8000000,70},{9000000,67},{10000000,81},{20000000,221},{30000000,62},{40000000,40},{50000000,17},{60000000,1},{70000000,1}", "size":"{131071,3969}"}

{"t":"1639163735", "op":"write", "count":"4159", "avgLatency":"3195912", "throughput":"272171008", "latency":"{500000,3},{600000,11},{700000,46},{800000,79},{900000,160},{1000000,174},{2000000,1751},{3000000,769},{4000000,346},{5000000,170},{6000000,108},{7000000,75},{8000000,52},{9000000,59},{10000000,38},{20000000,137},{30000000,51},{40000000,8},{50000000,10}", "size":"{131071,4038}"}

$ sudo estat iscsi 10
12/10/21 - 19:15:12 UTC

 Tracing enabled... Hit Ctrl-C to end.
   microseconds                                                     read
value range                 count ------------- Distribution ------------- 
[50, 60)                        1 |@                                       
[60, 70)                        6 |@                                       
[70, 80)                        1 |@                                       
[80, 90)                        3 |@                                       
[90, 100)                       1 |@                                       
[100, 200)                    168 |@                                       
[200, 300)                    401 |@                                       
[300, 400)                    556 |@                                       
[400, 500)                   1209 |@@                                      
[500, 600)                   1952 |@@                                      
[600, 700)                   2568 |@@@                                     
[700, 800)                   2788 |@@@                                     
[800, 900)                   2762 |@@@                                     
[900, 1000)                  2711 |@@@                                     
[1000, 2000)                16555 |@@@@@@@@@@@@@@@                         
[2000, 3000)                 5768 |@@@@@                                   
[3000, 4000)                 2070 |@@                                      
[4000, 5000)                 1062 |@                                       
[5000, 6000)                  811 |@                                       
[6000, 7000)                  886 |@                                       
[7000, 8000)                  656 |@                                       
[8000, 9000)                  545 |@                                       
[9000, 10000)                 456 |@                                       
[10000, 20000)               1774 |@@                                      
[20000, 30000)                579 |@                                       
[30000, 40000)                182 |@                                       
[40000, 50000)                 63 |@                                       
[50000, 60000)                 56 |@                                       
[60000, 70000)                 24 |@                                       
[70000, 80000)                 14 |@                                       
[80000, 90000)                  3 |@                                       
[90000, 100000)                 2 |@                                       
[100000, 200000)               11 |@                                       

   microseconds                                                    write
value range                 count ------------- Distribution ------------- 
[400, 500)                     44 |@                                       
[500, 600)                    308 |@                                       
[600, 700)                    858 |@                                       
[700, 800)                   1768 |@@                                      
[800, 900)                   2450 |@@@                                     
[900, 1000)                  2954 |@@@                                     
[1000, 2000)                23513 |@@@@@@@@@@@@@@@@@@@@@                   
[2000, 3000)                 6351 |@@@@@@                                  
[3000, 4000)                 1925 |@@                                      
[4000, 5000)                 1060 |@                                       
[5000, 6000)                  658 |@                                       
[6000, 7000)                  497 |@                                       
[7000, 8000)                  410 |@                                       
[8000, 9000)                  370 |@                                       
[9000, 10000)                 408 |@                                       
[10000, 20000)               2227 |@@                                      
[20000, 30000)                586 |@                                       
[30000, 40000)                204 |@                                       
[40000, 50000)                 63 |@                                       
[50000, 60000)                 38 |@                                       
[60000, 70000)                 60 |@                                       
[70000, 80000)                 15 |@                                       
[90000, 100000)                 3 |@                                       
[100000, 200000)                3 |@                                       
[200000, 300000)               64 |@                                       

                                       iops(/s)  avg latency(us)       stddev(us)  throughput(k/s)
read                                       4664             2968         25795568           298521
write                                      4683             3453         27252713           299756


                                       iops(/s)  throughput(k/s)
total                                      9348           598278