service startup II

[andre-merzky]

This improves agent service startup, and allows to expose service endpoints to tasks.

This closes #2899 and #3062.

[andre-merzky]

This is ready for review.

[andre-merzky]

TODO AM: add example

[codecov]

Codecov Report

Attention: Patch coverage is 46.66667% with 176 lines in your changes missing coverage. Please review.

Project coverage is 42.84%. Comparing base (8be5cd2) to head (ee76d0a).

Files with missing lines	Patch %	Lines
src/radical/pilot/client.py	21.84%	93 Missing ⚠️
src/radical/pilot/task_manager.py	8.33%	22 Missing ⚠️
src/radical/pilot/agent/agent_0.py	80.19%	20 Missing ⚠️
src/radical/pilot/session.py	23.52%	13 Missing ⚠️
src/radical/pilot/task.py	42.10%	11 Missing ⚠️
src/radical/pilot/agent/launch_method/flux.py	0.00%	6 Missing ⚠️
src/radical/pilot/agent/executing/base.py	83.33%	4 Missing ⚠️
src/radical/pilot/raptor/master.py	0.00%	2 Missing ⚠️
src/radical/pilot/agent/executing/sleep.py	0.00%	1 Missing ⚠️
src/radical/pilot/agent/scheduler/continuous.py	66.66%	1 Missing ⚠️
... and 3 more

Additional details and impacted files

@@            Coverage Diff             @@
##            devel    #3208      +/-   ##
==========================================
- Coverage   43.51%   42.84%   -0.67%     
==========================================
  Files          96       97       +1     
  Lines       10968    11167     +199     
==========================================
+ Hits         4773     4785      +12     
- Misses       6195     6382     +187     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[andre-merzky]

TODO AM: add example

Done

[andre-merzky]

Note: service specs on the pilot description still work?

[AymenFJA]

I tested this work on two machines :

• Local server, and it passed successfully.
• UVA Rivanna, and it passed successfully

RADICAL-Stack:

(base) jovyan@34a75d890b57:~$ radical-stack

  python               : /opt/conda/bin/python3
  pythonpath           : 
  version              : 3.9.13
  virtualenv           : base

  radical.analytics    : 1.60.0
  radical.entk         : 1.60.0
  radical.gtod         : 1.81.0
  radical.pilot        : 1.82.0-v1.81.0-53-g4c5009f05@feature/service_startup_2
  radical.utils        : 1.81.0

(base) jovyan@34a75d890b57:~$ 

For Local Server:

(base) jovyan@34a75d890b57:~/work/radical.pilot$ python examples/misc/service_tasks.py 
{'0': '12345'}
found my_service: {'0': '12345'}
task.000000: b'foo'

task.000001: b'foo'

task.000002: b'foo'

task.000003: b'foo'

For Rivanna:

(rct2) -bash-4.4$cat rp-submit.slurm-64529125.*
{'0': '12345'}
found my_service: {'0': '12345'}
task.000000: 
task.000001: 
task.000002: 
task.000003: 
(rct2) -bash-4.4$hostname
udc-ba38-32c0

[andre-merzky]

Does Rivanna allow that port to be opened and used? Any indication in the server logs or task logs on what happened?
The example is very simplistic obviously...

Note from call: we don't expect that simple example to work universally.