More robust gRPC connections, using tenacity where possible

[MaxiBoether]

Sometimes, we face outages/random disconnections during training. This fixes it in places where I encountered it last night. I tried to integrate tenacity as suggested by @robinholzi, but it's not always possible since the retry logic involves keeping track of already done work, which I don't want to put into class state

Part 6/n of porting over SIGMOD changes.

[robinholzi]

@MaxiBoether Not fully what the problem with partial result states is, but have you considered using retry contexts?

[MaxiBoether]

@MaxiBoether Not fully what the problem with partial result states is, but have you considered using retry contexts?

Ah, nope. I guess I can refactor the code to retry contexts then

[codecov]

Codecov Report

Attention: Patch coverage is 71.59091% with 25 lines in your changes missing coverage. Please review.

Project coverage is 82.50%. Comparing base (75868bb) to head (ac1d863).

❗ Current head ac1d863 differs from pull request most recent head 19cde94

Please upload reports for the commit 19cde94 to get more accurate results.

Files	Patch %	Lines
...n/evaluator/internal/dataset/evaluation_dataset.py	61.11%	14 Missing ⚠️
modyn/supervisor/internal/grpc_handler.py	50.00%	5 Missing ⚠️
.../internal/pipeline_executor/evaluation_executor.py	54.54%	5 Missing ⚠️
.../trainer_server/internal/dataset/online_dataset.py	95.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #521      +/-   ##
==========================================
- Coverage   82.55%   82.50%   -0.05%     
==========================================
  Files         214      214              
  Lines        9975    10039      +64     
==========================================
+ Hits         8235     8283      +48     
- Misses       1740     1756      +16     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[MaxiBoether]

@MaxiBoether Not fully what the problem with partial result states is, but have you considered using retry contexts?

Thanks for suggesting that. I adapted the code. However, I needed to manually catch and reraise exceptions sometimes because I find their callback system to be quite ugly in object-oriented contexts (we need to write a static function that recovers self out of args[0] etc). I think this actually looks better than using callbacks, and it allows for easier logging with our logging infra. The retry logic itself is still more hidden now :)

[robinholzi]

Thanks for the refactoring! Looks good now :)

[github-actions]

^{( % to main)
( % to main)}