server: fix deadlock of pending_session_id_rotations mutex

[kp-mariappan-ramasamy]

Description

Commit bccfc81 fixes the leaking UDP sessions. But it introduces a deadlock, by trying to lock the same Mutex twice consecutively on error condition.

Luckily our parking lot deadlock detection pin pointed it out.

Fixing it by locking once and using the same guard for removing stale entry.

Motivation and Context

Server instances are restarted frequently

How Has This Been Tested?

Verified in prod that instances are stable without deadlock

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

• My change requires a change to the documentation.
• I have updated the documentation accordingly.
• The correct base branch is being used, if not main

[github-actions]

Code coverage summary for 91eb6a8:

Filename                                                     Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
lightway-app-utils/src/args/cipher.rs                              4                 4     0.00%           1                 1     0.00%           5                 5     0.00%           0                 0         -
lightway-app-utils/src/args/connection_type.rs                     4                 4     0.00%           1                 1     0.00%           5                 5     0.00%           0                 0         -
lightway-app-utils/src/args/duration.rs                            7                 7     0.00%           5                 5     0.00%          13                13     0.00%           0                 0         -
lightway-app-utils/src/args/ip_map.rs                             11                11     0.00%           3                 3     0.00%          15                15     0.00%           0                 0         -
lightway-app-utils/src/args/logging.rs                            14                14     0.00%           2                 2     0.00%          16                16     0.00%           0                 0         -
lightway-app-utils/src/connection_ticker.rs                       71                11    84.51%          28                 4    85.71%         279                11    96.06%           0                 0         -
lightway-app-utils/src/dplpmtud_timer.rs                          63                 7    88.89%          22                 4    81.82%         282                11    96.10%           0                 0         -
lightway-app-utils/src/event_stream.rs                             5                 0   100.00%           3                 0   100.00%          13                 0   100.00%           0                 0         -
lightway-app-utils/src/iouring.rs                                157               142     9.55%          23                17    26.09%         304               281     7.57%           0                 0         -
lightway-app-utils/src/metrics.rs                                  2                 2     0.00%           2                 2     0.00%           4                 4     0.00%           0                 0         -
lightway-app-utils/src/sockopt/ip_mtu_discover.rs                 23                23     0.00%           4                 4     0.00%          93                93     0.00%           0                 0         -
lightway-app-utils/src/sockopt/ip_pktinfo.rs                       4                 4     0.00%           1                 1     0.00%          16                16     0.00%           0                 0         -
lightway-app-utils/src/tun.rs                                     76                76     0.00%          22                22     0.00%          94                94     0.00%           0                 0         -
lightway-app-utils/src/utils.rs                                   13                13     0.00%           1                 1     0.00%          11                11     0.00%           0                 0         -
lightway-client/src/args.rs                                       35                35     0.00%          31                31     0.00%          38                38     0.00%           0                 0         -
lightway-client/src/io/inside/tun.rs                              39                39     0.00%           7                 7     0.00%          48                48     0.00%           0                 0         -
lightway-client/src/io/outside/tcp.rs                             48                48     0.00%          10                10     0.00%          45                45     0.00%           0                 0         -
lightway-client/src/io/outside/udp.rs                             62                62     0.00%          12                12     0.00%          62                62     0.00%           0                 0         -
lightway-client/src/keepalive.rs                                 245                33    86.53%          56                 8    85.71%         372                26    93.01%           0                 0         -
lightway-client/src/lib.rs                                       257               257     0.00%          27                27     0.00%         383               383     0.00%           0                 0         -
lightway-client/src/main.rs                                       48                48     0.00%           7                 7     0.00%         169               169     0.00%           0                 0         -
lightway-core/src/borrowed_bytesmut.rs                            86                 1    98.84%          24                 0   100.00%         196                 1    99.49%           0                 0         -
lightway-core/src/builder_predicates.rs                           20                10    50.00%           4                 2    50.00%          28                14    50.00%           0                 0         -
lightway-core/src/cipher.rs                                        8                 0   100.00%           3                 0   100.00%          13                 0   100.00%           0                 0         -
lightway-core/src/connection.rs                                  671               404    39.79%          58                27    53.45%         877               465    46.98%           0                 0         -
lightway-core/src/connection/builders.rs                          67                25    62.69%          22                11    50.00%         260                56    78.46%           0                 0         -
lightway-core/src/connection/dplpmtud.rs                         761                90    88.17%          66                 1    98.48%         947                11    98.84%           0                 0         -
lightway-core/src/connection/fragment_map.rs                     132                11    91.67%          27                 0   100.00%         290                 7    97.59%           0                 0         -
lightway-core/src/connection/io_adapter.rs                       137                17    87.59%          36                 5    86.11%         313                23    92.65%           0                 0         -
lightway-core/src/connection/key_update.rs                        23                 7    69.57%           5                 0   100.00%          38                19    50.00%           0                 0         -
lightway-core/src/context.rs                                      91                27    70.33%          26                 9    65.38%         213                52    75.59%           0                 0         -
lightway-core/src/context/ip_pool.rs                               7                 3    57.14%           1                 0   100.00%           6                 0   100.00%           0                 0         -
lightway-core/src/context/server_auth.rs                          14                11    21.43%           4                 3    25.00%          24                20    16.67%           0                 0         -
lightway-core/src/io.rs                                            3                 3     0.00%           3                 3     0.00%          10                10     0.00%           0                 0         -
lightway-core/src/lib.rs                                           9                 6    33.33%           4                 1    75.00%          18                 9    50.00%           0                 0         -
lightway-core/src/metrics.rs                                      17                17     0.00%          15                15     0.00%          33                33     0.00%           0                 0         -
lightway-core/src/packet.rs                                       27                 7    74.07%           4                 1    75.00%          30                 6    80.00%           0                 0         -
lightway-core/src/plugin.rs                                       66                 8    87.88%          23                 5    78.26%         160                 9    94.38%           0                 0         -
lightway-core/src/utils.rs                                       101                21    79.21%          22                 2    90.91%         184                17    90.76%           0                 0         -
lightway-core/src/version.rs                                      37                 0   100.00%          19                 0   100.00%          89                 0   100.00%           0                 0         -
lightway-core/src/wire.rs                                        156                27    82.69%          32                 0   100.00%         267                 6    97.75%           0                 0         -
lightway-core/src/wire/auth_failure.rs                             9                 1    88.89%           3                 0   100.00%          19                 0   100.00%           0                 0         -
lightway-core/src/wire/auth_request.rs                           147                12    91.84%          34                 0   100.00%         287                 0   100.00%           0                 0         -
lightway-core/src/wire/auth_success_with_config_ipv4.rs           69                 4    94.20%          13                 0   100.00%         145                 0   100.00%           0                 0         -
lightway-core/src/wire/data.rs                                    20                 1    95.00%           7                 0   100.00%          43                 0   100.00%           0                 0         -
lightway-core/src/wire/data_frag.rs                               41                 1    97.56%          17                 0   100.00%          92                 0   100.00%           0                 0         -
lightway-core/src/wire/encoding_request.rs                        22                 2    90.91%           6                 0   100.00%          44                 1    97.73%           0                 0         -
lightway-core/src/wire/encoding_response.rs                       22                 2    90.91%           6                 0   100.00%          44                 1    97.73%           0                 0         -
lightway-core/src/wire/ping.rs                                    22                 3    86.36%           7                 0   100.00%          68                 0   100.00%           0                 0         -
lightway-core/src/wire/pong.rs                                    15                 2    86.67%           5                 0   100.00%          34                 0   100.00%           0                 0         -
lightway-core/src/wire/server_config.rs                           20                 2    90.00%           5                 0   100.00%          44                 0   100.00%           0                 0         -
lightway-server/src/args.rs                                       30                30     0.00%          30                30     0.00%          30                30     0.00%           0                 0         -
lightway-server/src/auth.rs                                      104                36    65.38%          19                 5    73.68%         148                28    81.08%           0                 0         -
lightway-server/src/connection.rs                                 57                57     0.00%          25                25     0.00%         123               123     0.00%           0                 0         -
lightway-server/src/connection_manager.rs                         81                81     0.00%          24                24     0.00%         219               219     0.00%           0                 0         -
lightway-server/src/connection_manager/connection_map.rs          87                 8    90.80%          26                 2    92.31%         273                 8    97.07%           0                 0         -
lightway-server/src/io/inside/tun.rs                              25                25     0.00%           8                 8     0.00%          34                34     0.00%           0                 0         -
lightway-server/src/io/outside/tcp.rs                             80                80     0.00%           8                 8     0.00%          99                99     0.00%           0                 0         -
lightway-server/src/io/outside/udp.rs                            144               144     0.00%          18                18     0.00%         263               263     0.00%           0                 0         -
lightway-server/src/io/outside/udp/cmsg.rs                        47                16    65.96%          14                 5    64.29%         237                56    76.37%           0                 0         -
lightway-server/src/ip_manager.rs                                 82                25    69.51%          19                 4    78.95%         254                28    88.98%           0                 0         -
lightway-server/src/ip_manager/ip_pool.rs                        101                 1    99.01%          32                 0   100.00%         303                 0   100.00%           0                 0         -
lightway-server/src/lib.rs                                       122               122     0.00%          14                14     0.00%         198               198     0.00%           0                 0         -
lightway-server/src/main.rs                                       81                81     0.00%          10                10     0.00%         217               217     0.00%           0                 0         -
lightway-server/src/metrics.rs                                    93                91     2.15%          82                80     2.44%         220               216     1.82%           0                 0         -
lightway-server/src/statistics.rs                                 53                23    56.60%           8                 4    50.00%         103                42    59.22%           0                 0         -
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                           5195              2385    54.09%        1106               491    55.61%        9804              3667    62.60%           0                 0         -

✅ Region coverage 54% passes
✅ Line coverage 62% passes