Skip to content

Issue 182 - Error handling, propagation, etc. #187

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 23 commits into from
Apr 25, 2024
Merged

Issue 182 - Error handling, propagation, etc. #187

merged 23 commits into from
Apr 25, 2024

Conversation

huntharo
Copy link
Contributor

@huntharo huntharo commented Apr 24, 2024 •
edited
Loading

  • ENHANCEMENT: Move contained app response relay to a task and await both the request and response relay tasks
  • FIX: Tell other channels to stop looping when 1 channel stops looping
  • ENHANCEMENT: Propagate detailed errors and exit reasons all the way back to the router
  • FIX/ENHANCEMENT: Remove hiding of errors (non-consumed results)
  • ENHANCEMENT: Handle Channel 5xx, 4xx, and Other ExitReasons

Fixes #182

@huntharo huntharo self-assigned this Apr 24, 2024
@github-actions GitHub Actions
Copy link

github-actions bot commented Apr 24, 2024 •
edited
Loading

📊 DotNet Code Coverage Report

File Lines Lines Hit / Found Uncovered Lines Branches
/src/PwrDrvr.LambdaDisp..ension/MetricsRegistry.cs ❌ 0.0% 0 / 21 21 0.0%
/src/PwrDrvr.LambdaDisp..n/HttpReverseRequester.cs ❌ 0.0% 0 / 41 41 0.0%
/src/PwrDrvr.LambdaDispatch.Extension/Function.cs ❌ 0.0% 0 / 462 462 0.0%
/src/PwrDrvr.LambdaDisp..sion/HttpDuplexContent.cs ❌ 0.0% 0 / 23 23 0.0%
/src/PwrDrvr.LambdaDisp..n/HttpReverseRequester.cs ❌ 0.0% 0 / 296 296 0.0%
/src/PwrDrvr.LambdaDisp..tension/LoggerInstance.cs ❌ 0.0% 0 / 17 17 -
/src/PwrDrvr.LambdaDisp..ension/MetricsRegistry.cs ❌ 0.0% 0 / 21 21 -
/src/PwrDrvr.LambdaDisp..ension/MetricsRegistry.cs ❌ 0.0% 0 / 71 71 0.0%
/src/PwrDrvr.LambdaDisp..on/TcpReverseRequester.cs ❌ 0.0% 0 / 194 194 0.0%
/src/PwrDrvr.LambdaDisp..ension/SetupHttpClient.cs ❌ 0.0% 0 / 33 33 0.0%
/src/PwrDrvr.LambdaDispatch.Messages/Waiter.cs ✅ 100.0% 6 / 6 0 -
/src/PwrDrvr.LambdaDispatch.Messages/Waiter.cs ❌ 0.0% 0 / 2 2 -
/src/PwrDrvr.LambdaDisp...Router/LambdaInstance.cs ✅ 100.0% 3 / 3 0 -
/src/PwrDrvr.LambdaDisp..Router/CapacityManager.cs ✅ 100.0% 51 / 51 0 100.0%
/src/PwrDrvr.LambdaDisp..Router/MetricsRegistry.cs ❌ 2.8% 1 / 36 35 0.0%
/src/PwrDrvr.LambdaDispatch.Router/Config.cs ✅ 98.9% 94 / 95 1 97.7%
/src/PwrDrvr.LambdaDisp..lers/ChunkedController.cs 🟡 88.3% 121 / 137 16 91.7%
/src/PwrDrvr.LambdaDisp..lers/DefaultController.cs ✅ 100.0% 5 / 5 0 -
/src/PwrDrvr.LambdaDisp..ollers/ResetController.cs ✅ 100.0% 8 / 8 0 -
/src/PwrDrvr.LambdaDispatch.Router/Dispatcher.cs 🟡 81.6% 248 / 304 56 71.0%
/src/PwrDrvr.LambdaDispatch.Router/Dispatcher.cs ✅ 100.0% 3 / 3 0 -
/src/PwrDrvr.LambdaDisp..dMetrics/MetricsLogger.cs 🟡 80.9% 114 / 141 27 80.0%
/src/PwrDrvr.LambdaDisp..ics/MetricsLoggerDummy.cs ❌ 0.0% 0 / 2 2 -
/src/PwrDrvr.LambdaDisp..h.Router/GetCallbackIP.cs ✅ 100.0% 8 / 8 0 100.0%
/src/PwrDrvr.LambdaDisp..omingRequestMiddleware.cs ✅ 100.0% 21 / 21 0 83.3%
/src/PwrDrvr.LambdaDisp..Router/LambdaArnParser.cs ✅ 94.6% 35 / 37 2 87.5%
/src/PwrDrvr.LambdaDisp..ter/LambdaClientConfig.cs ✅ 100.0% 18 / 18 0 100.0%
/src/PwrDrvr.LambdaDisp..outer/LambdaConnection.cs ❌ 31.1% 95 / 305 210 10.7%
/src/PwrDrvr.LambdaDisp...Router/LambdaInstance.cs 🟡 59.9% 239 / 399 160 47.5%
/src/PwrDrvr.LambdaDisp../LambdaInstanceManager.cs ✅ 100.0% 4 / 4 0 -
/src/PwrDrvr.LambdaDisp../LambdaInstanceManager.cs ❌ 35.1% 152 / 433 281 64.3%
/src/PwrDrvr.LambdaDisp../LeastOutstandingQueue.cs 🟡 67.4% 145 / 215 70 65.6%
/src/PwrDrvr.LambdaDisp...Router/LoggerInstance.cs ✅ 100.0% 18 / 18 0 -
/src/PwrDrvr.LambdaDisp..Router/MetricsRegistry.cs ❌ 47.6% 10 / 21 11 -
/src/PwrDrvr.LambdaDisp..Router/MetadataService.cs 🟡 84.4% 54 / 64 10 67.6%
/src/PwrDrvr.LambdaDisp..Router/MetricsRegistry.cs ✅ 91.2% 124 / 136 12 -
/src/PwrDrvr.LambdaDisp..Router/MetricsRegistry.cs ❌ 0.0% 0 / 31 31 -
/src/PwrDrvr.LambdaDisp...Router/PendingRequest.cs ✅ 90.0% 45 / 50 5 66.7%
/src/PwrDrvr.LambdaDispatch.Router/Pool.cs ✅ 100.0% 7 / 7 0 -
/src/PwrDrvr.LambdaDispatch.Router/PoolManager.cs 🟡 89.4% 59 / 66 7 78.6%
/src/PwrDrvr.LambdaDispatch.Router/PoolOptions.cs ✅ 100.0% 9 / 9 0 100.0%
/src/PwrDrvr.LambdaDispatch.Router/Program.cs ❌ 0.0% 0 / 157 157 0.0%
/src/PwrDrvr.LambdaDisp..binLambdaInstanceQueue.cs ✅ 90.6% 29 / 32 3 91.7%
/src/PwrDrvr.LambdaDisp..inLambdaInstanceQueue2.cs ✅ 95.3% 61 / 64 3 100.0%
/src/PwrDrvr.LambdaDisp...Router/ShutdownSignal.cs ❌ 0.0% 0 / 1 1 -
/src/PwrDrvr.LambdaDispatch.Router/Startup.cs ❌ 0.0% 0 / 47 47 0.0%
/src/PwrDrvr.LambdaDispatch.Router/TokenBucket.cs 🟡 66.7% 12 / 18 6 100.0%
/src/PwrDrvr.LambdaDisp...Router/LambdaInstance.cs ✅ 100.0% 3 / 3 0 -
/src/PwrDrvr.LambdaDisp..Router/WeightedAverage.cs ✅ 91.7% 44 / 48 4 100.0%
/src/PwrDrvr.LambdaDisp..Router/TrailingAverage.cs ✅ 100.0% 36 / 36 0 100.0%
Overall ❌ 44.6% 1882 / 4220 2338 44.6%

huntharo added 18 commits April 24, 2024 16:46
@huntharo
Issue-182 - connect_to_app test fixes
db55f1b
@huntharo
Issue-182 - LambdaRequest tests
4e0eb98
@huntharo
Issue-182 - RouterChannel remove test comments
7559161
@huntharo
Issue-182 - LambdaService error test
d96061d
@huntharo
Issue-182 - Implement RouterUnreachable error
8b3e854
@huntharo
Issue-182 - test_mock_router
dd39bb8
@huntharo
Issue-182 - RouterChannel FIXME comments
6c044f4
@huntharo
Issue-182 - RouterChannel exit failing test
ef9c076
@huntharo
Issue-182 - RouterChannel response panics
0d9729c
@huntharo
Issue-182 - RouterChannel fix not exiting on goaway set
508ff7a 
- RouterChannel loop was not checking the goaway signal at all
- As a result, the RouterChannel would just loop as long as it kept getting requests if the ping task failed and exited
@huntharo
Issue-182 - Simplify rps formatting
e3bfb19
@huntharo
Issue-182 - RouterChannel error propagation
5032b0a 
- Move the app response relay into it's own task
   - This allows exiting that task on error and returning the error
- Translate errors into LambdaRequestError
- Return the worst error from RouterChannel if an error happened
- Replace `?;` and `.unwrap()` that can fail with `.map_err(|_| LambdaRequestError::[some])?;`
@huntharo
Issue-182 - Squelch log spew on healthcheck
4461624
@huntharo
Issue-182 - Add exit reasons and error types
2c23fd4
@huntharo
Issue-182 - LambdaService catch invalid lambda payload
700b23d 
- Invalid payload handling
- Add strongly typed request validation and error codes
- Add tests for sent_time and router_url being invalid
- RouterChannel - Change failure to connect to contained app to `AppConnectionUnreachable` error
@huntharo
Issue-182 - Add comment to main about how to reproduce a fatal error
e71de92
@huntharo
Issue-182 - LambdaService add AppConnectionUnreachable fatal error test
5d3c559
@huntharo
Issue-182 - Improve test
bf6f894
@huntharo huntharo force-pushed the issue-182/error-handling branch from 66e97b5 to bf6f894 Compare April 24, 2024 20:46
@github-actions GitHub Actions
Copy link

github-actions bot commented Apr 24, 2024 •
edited
Loading

📊 Rust Code Coverage Report

File Lines Lines Hit / Found Uncovered Lines Branches
/extension/src/app_request.rs ✅ 98.7% 1004 / 1017 13 -
/extension/src/app_start.rs 🟡 85.9% 225 / 262 37 -
/extension/src/cert.rs 🟡 88.3% 188 / 213 25 -
/extension/src/connect_to_app.rs 🟡 88.8% 239 / 269 30 -
/extension/src/connect_to_router.rs ✅ 92.3% 298 / 323 25 -
/extension/src/counter_drop.rs ✅ 100.0% 23 / 23 0 -
/extension/src/endpoint.rs ✅ 93.2% 124 / 133 9 -
/extension/src/lambda_request.rs ✅ 92.3% 512 / 555 43 -
/extension/src/lambda_request_error.rs 🟡 78.1% 100 / 128 28 -
/extension/src/lambda_service.rs 🟡 88.6% 2128 / 2402 274 -
/extension/src/main.rs 🟡 83.7% 251 / 300 49 -
/extension/src/messages.rs ✅ 91.8% 235 / 256 21 -
/extension/src/options.rs ✅ 100.0% 154 / 154 0 -
/extension/src/ping.rs 🟡 89.6% 1003 / 1120 117 -
/extension/src/router_channel.rs 🟡 89.1% 1137 / 1276 139 -
/extension/src/test_http2_server.rs ✅ 96.2% 76 / 79 3 -
/extension/src/test_mock_router.rs ✅ 90.2% 138 / 153 15 -
/extension/src/threads.rs 🟡 71.4% 5 / 7 2 -
/extension/src/time.rs ✅ 100.0% 6 / 6 0 -
Overall ✅ 90.4% 7846 / 8676 830 -

@huntharo huntharo mentioned this pull request Apr 24, 2024
Closed
4 tasks
huntharo
huntharo commented Apr 24, 2024
View reviewed changes
huntharo
huntharo commented Apr 24, 2024
View reviewed changes
huntharo
huntharo commented Apr 24, 2024
View reviewed changes
huntharo
huntharo commented Apr 24, 2024
View reviewed changes
huntharo
huntharo commented Apr 24, 2024
View reviewed changes
huntharo
huntharo commented Apr 24, 2024
View reviewed changes
huntharo
huntharo commented Apr 24, 2024
View reviewed changes
huntharo
huntharo commented Apr 24, 2024
View reviewed changes
huntharo
huntharo commented Apr 24, 2024
View reviewed changes
huntharo
huntharo commented Apr 25, 2024
View reviewed changes
@huntharo huntharo merged commit 536f468 into main Apr 25, 2024
@huntharo huntharo deleted the issue-182/error-handling branch April 25, 2024 03:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@huntharo huntharo

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Extension - Error handling improvements and tests
1 participant
@huntharo