jmp shell / pytest lease handling

[mangelajo]

The lease waiting/handling mechanism needs several improvements:

• The timeout waiting for a lease should be configurable
• The lease waiting progress should be informative, not just waiting in silence.
• If timeout happens, a real error message should be provided instead of a huge traceback, this is specially noticeable when using the pytest class.

$ jmp shell -l name=${JETSON_HARDWARE_RUNNER} --duration 01:00:00 ./scripts/testing/test-rhel-bootc-via-new-jumpstarter.sh ${BOOTC_TEST_IMAGE} ${JETSON_HARDWARE_RUNNER}
[09/29/25 10:10:27] INFO     INFO:jumpstarter.client.lease:Created   lease.py:60
                             lease request for selector                         
                             name=nvidia-jetson-nx-orin-01-khw-eng-b            
                             os2-beaker for duration 1:00:00                    
╭───────────────────── Traceback (most recent call last) ──────────────────────╮
│ /tmp/.local/jumpstarter/venv/lib64/python3.12/site-packages/anyio/_core/_tas │
│ ks.py:120 in fail_after                                                      │
│                                                                              │
│   117 │   with get_async_backend().create_cancel_scope(                      │
│   118 │   │   deadline=deadline, shield=shield                               │
│   119 │   ) as cancel_scope:                                                 │
│ ❱ 120 │   │   yield cancel_scope                                             │
│   121 │                                                                      │
│   122 │   if cancel_scope.cancelled_caught and current_time() >= cancel_scop │
│   123 │   │   raise TimeoutError                                             │
│                                                                              │
│ /tmp/.local/jumpstarter/venv/lib64/python3.12/site-packages/jumpstarter/clie │
│ nt/lease.py:137 in _acquire                                                  │
│                                                                              │
│   134 │   │   │   │   if condition_present_and_equal(result.conditions, "Rea │
│   135 │   │   │   │   │   raise LeaseError(f"lease {self.name} released")    │
│   136 │   │   │   │                                                          │
│ ❱ 137 │   │   │   │   await sleep(1)                                         │
│   138 │                                                                      │
│   139 │   @asynccontextmanager                                               │
│   140 │   async def __asynccontextmanager__(self) -> AsyncGenerator[Self]:   │
│                                                                              │
│ /tmp/.local/jumpstarter/venv/lib64/python3.12/site-packages/anyio/_core/_eve │
│ ntloop.py:87 in sleep                                                        │
│                                                                              │
│    84 │   :param delay: the duration, in seconds                             │
│    85 │                                                                      │
│    86 │   """                                                                │
│ ❱  87 │   return await get_async_backend().sleep(delay)                      │
│    88                                                                        │
│    89                                                                        │
│    90 async def sleep_forever() -> None:                                     │
│                                                                              │
│ /tmp/.local/jumpstarter/venv/lib64/python3.12/site-packages/anyio/_backends/ │
│ _asyncio.py:2369 in sleep                                                    │
│                                                                              │
│   2366 │                                                                     │
│   2367 │   @classmethod                                                      │
│   2368 │   async def sleep(cls, delay: float) -> None:                       │
│ ❱ 2369 │   │   await sleep(delay)                                            │
│   2370 │                                                                     │
│   2371 │   @classmethod                                                      │
│   2372 │   def create_cancel_scope(                                          │
│                                                                              │
│ /usr/lib64/python3.12/asyncio/tasks.py:665 in sleep                          │
│                                                                              │
│    662 │   │   │   │   │   │   futures._set_result_unless_cancelled,         │
│    663 │   │   │   │   │   │   future, result)                               │
│    664 │   try:                                                              │
│ ❱  665 │   │   return await future                                           │
│    666 │   finally:                                                          │
│    667 │   │   h.cancel()                                                    │
│    668                                                                       │
╰──────────────────────────────────────────────────────────────────────────────╯
CancelledError: Cancelled via cancel scope ffffa76d6660; reason: deadline 
exceeded
During handling of the above exception, another exception occurred:
╭───────────────────── Traceback (most recent call last) ──────────────────────╮
│ /tmp/.local/jumpstarter/venv/bin/jmp:8 in <module>                           │
│                                                                              │
│   5 from jumpstarter_cli.jmp import jmp                                      │
│   6 if __name__ == '__main__':                                               │
│   7 │   sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])     │
│ ❱ 8 │   sys.exit(jmp())                                                      │
│   9                                                                          │
│                                                                              │
│ /tmp/.local/jumpstarter/venv/lib64/python3.12/site-packages/click/core.py:14 │
│ 62 in __call__                                                               │
│                                                                              │
│   1459 │                                                                     │
│   1460 │   def __call__(self, *args: t.Any, **kwargs: t.Any) -> t.Any:       │
│   1461 │   │   """Alias for :meth:`main`."""                                 │
│ ❱ 1462 │   │   return self.main(*args, **kwargs)                             │
│   1463                                                                       │
│   1464                                                                       │
│   1465 class _FakeSubclassCheck(type):                                       │
│                                                                              │
│ /tmp/.local/jumpstarter/venv/lib64/python3.12/site-packages/click/core.py:13 │
│ 83 in main                                                                   │
│                                                                              │
│   1380 │   │   try:                                                          │
│   1381 │   │   │   try:                                                      │
│   1382 │   │   │   │   with self.make_context(prog_name, args, **extra) as c │
│ ❱ 1383 │   │   │   │   │   rv = self.invoke(ctx)                             │
│   1384 │   │   │   │   │   if not standalone_mode:                           │
│   1385 │   │   │   │   │   │   return rv                                     │
│   1386 │   │   │   │   │   # it's not safe to `ctx.exit(rv)` here!           │
│                                                                              │
│ /tmp/.local/jumpstarter/venv/lib64/python3.12/site-packages/click/core.py:18 │
│ 50 in invoke                                                                 │
│                                                                              │
│   1847 │   │   │   │   super().invoke(ctx)                                   │
│   1848 │   │   │   │   sub_ctx = cmd.make_context(cmd_name, args, parent=ctx │
│   1849 │   │   │   │   with sub_ctx:                                         │
│ ❱ 1850 │   │   │   │   │   return _process_result(sub_ctx.command.invoke(sub │
│   1851 │   │                                                                 │
│   1852 │   │   # In chain mode we create the contexts step by step, but afte │
│   1853 │   │   # base command has been invoked.  Because at that point we do │
│                                                                              │
│ /tmp/.local/jumpstarter/venv/lib64/python3.12/site-packages/click/core.py:12 │
│ 46 in invoke                                                                 │
│                                                                              │
│   1243 │   │   │   echo(style(message, fg="red"), err=True)                  │
│   1244 │   │                                                                 │
│   1245 │   │   if self.callback is not None:                                 │
│ ❱ 1246 │   │   │   return ctx.invoke(self.callback, **ctx.params)            │
│   1247 │                                                                     │
│   1248 │   def shell_complete(self, ctx: Context, incomplete: str) -> list[C │
│   1249 │   │   """Return a list of completions for the incomplete value. Loo │
│                                                                              │
│ /tmp/.local/jumpstarter/venv/lib64/python3.12/site-packages/click/core.py:81 │
│ 4 in invoke                                                                  │
│                                                                              │
│    [811](https://gitlab.com/redhat/rhel/sst/orin-sidecar/nvidia-jetson-sidecar/-/jobs/11531929594#L811) │   │                                                                 │
│    [812](https://gitlab.com/redhat/rhel/sst/orin-sidecar/nvidia-jetson-sidecar/-/jobs/11531929594#L812) │   │   with augment_usage_errors(self):                              │
│    [813](https://gitlab.com/redhat/rhel/sst/orin-sidecar/nvidia-jetson-sidecar/-/jobs/11531929594#L813) │   │   │   with ctx:                                                 │
│ ❱  [814](https://gitlab.com/redhat/rhel/sst/orin-sidecar/nvidia-jetson-sidecar/-/jobs/11531929594#L814) │   │   │   │   return callback(*args, **kwargs)                      │
│    [815](https://gitlab.com/redhat/rhel/sst/orin-sidecar/nvidia-jetson-sidecar/-/jobs/11531929594#L815) │                                                                     │
│    [816](https://gitlab.com/redhat/rhel/sst/orin-sidecar/nvidia-jetson-sidecar/-/jobs/11531929594#L816) │   def forward(self, cmd: Command, /, *args: t.Any, **kwargs: t.Any) │
│    [817](https://gitlab.com/redhat/rhel/sst/orin-sidecar/nvidia-jetson-sidecar/-/jobs/11531929594#L817) │   │   """Similar to :meth:`invoke` but fills in default keyword     │
│                                                                              │
│ /tmp/.local/jumpstarter/venv/lib64/python3.12/site-packages/jumpstarter_cli_ │
│ common/config.py:94 in wrapper                                               │
│                                                                              │
│    91 │   │   except Exception as e:                                         │
│    92 │   │   │   raise click.ClickException("Failed to load config: {}".for │
│    93 │   │                                                                  │
│ ❱  94 │   │   return f(*args, **kwds, config=config)                         │
│    95 │                                                                      │
│    96 │   return reduce(lambda w, opt: opt(w), options, wrapper)             │
│    97                                                                        │
│                                                                              │
│ /tmp/.local/jumpstarter/venv/lib64/python3.12/site-packages/jumpstarter_cli_ │
│ common/exceptions.py:56 in wrapped                                           │
│                                                                              │
│    53 │   │   @wraps(func)                                                   │
│    54 │   │   def wrapped(*args, **kwargs):                                  │
│    55 │   │   │   try:                                                       │
│ ❱  56 │   │   │   │   return func(*args, **kwargs)                           │
│    57 │   │   │   except ConnectionError as e:                               │
│    58 │   │   │   │   if "expired" in str(e).lower():                        │
│    59 │   │   │   │   │   click.echo(click.style("Token is expired, triggeri │
│                                                                              │
│ /tmp/.local/jumpstarter/venv/lib64/python3.12/site-packages/jumpstarter_cli/ │
│ shell.py:52 in shell                                                         │
│                                                                              │
│   49 │   │   │   │   │   command=command,                                    │
│   50 │   │   │   │   )                                                       │
│   51 │   │   │                                                               │
│ ❱ 52 │   │   │   with config.lease(selector=selector, lease_name=lease_name, │
│   53 │   │   │   │   with lease.serve_unix() as path:                        │
│   54 │   │   │   │   │   with lease.monitor():                               │
│   55 │   │   │   │   │   │   if exporter_logs:                               │
│                                                                              │
│ /usr/lib64/python3.12/contextlib.py:137 in __enter__                         │
│                                                                              │
│   134 │   │   # they are only needed for recreation, which is not possible a │
│   135 │   │   del self.args, self.kwds, self.func                            │
│   136 │   │   try:                                                           │
│ ❱ 137 │   │   │   return next(self.gen)                                      │
│   138 │   │   except StopIteration:                                          │
│   139 │   │   │   raise RuntimeError("generator didn't yield") from None     │
│   140                                                                        │
│                                                                              │
│ /tmp/.local/jumpstarter/venv/lib64/python3.12/site-packages/jumpstarter/conf │
│ ig/client.py:130 in lease                                                    │
│                                                                              │
│   127 │   │   duration: timedelta = timedelta(minutes=30),                   │
│   128 │   ):                                                                 │
│   129 │   │   with start_blocking_portal() as portal:                        │
│ ❱ 130 │   │   │   with portal.wrap_async_context_manager(self.lease_async(se │
│   131 │   │   │   │   yield lease                                            │
│   132 │                                                                      │
│   133 │   @_blocking_compat                                                  │
│                                                                              │
│ /tmp/.local/jumpstarter/venv/lib64/python3.12/site-packages/anyio/from_threa │
│ d.py:155 in __enter__                                                        │
│                                                                              │
│   152 │   def __enter__(self) -> T_co:                                       │
│   153 │   │   self._enter_future = Future()                                  │
│   154 │   │   self._exit_future = self._portal.start_task_soon(self.run_asyn │
│ ❱ 155 │   │   return self._enter_future.result()                             │
│   156 │                                                                      │
│   157 │   def __exit__(                                                      │
│   158 │   │   self,                                                          │
│                                                                              │
│ /usr/lib64/python3.12/concurrent/futures/_base.py:456 in result              │
│                                                                              │
│   453 │   │   │   │   if self._state in [CANCELLED, CANCELLED_AND_NOTIFIED]: │
│   454 │   │   │   │   │   raise CancelledError()                             │
│   455 │   │   │   │   elif self._state == FINISHED:                          │
│ ❱ 456 │   │   │   │   │   return self.__get_result()                         │
│   457 │   │   │   │   else:                                                  │
│   458 │   │   │   │   │   raise TimeoutError()                               │
│   459 │   │   finally:                                                       │
│                                                                              │
│ /usr/lib64/python3.12/concurrent/futures/_base.py:401 in __get_result        │
│                                                                              │
│   398 │   def __get_result(self):                                            │
│   399 │   │   if self._exception:                                            │
│   400 │   │   │   try:                                                       │
│ ❱ 401 │   │   │   │   raise self._exception                                  │
│   402 │   │   │   finally:                                                   │
│   403 │   │   │   │   # Break a reference cycle with the exception in self._ │
│   404 │   │   │   │   self = None                                            │
│                                                                              │
│ /tmp/.local/jumpstarter/venv/lib64/python3.12/site-packages/anyio/from_threa │
│ d.py:252 in _call_func                                                       │
│                                                                              │
│   249 │   │   │   │   │   else:                                              │
│   250 │   │   │   │   │   │   future.add_done_callback(callback)             │
│   251 │   │   │   │   │                                                      │
│ ❱ 252 │   │   │   │   │   retval = await retval_or_awaitable                 │
│   253 │   │   │   else:                                                      │
│   254 │   │   │   │   retval = retval_or_awaitable                           │
│   255 │   │   except self._cancelled_exc_class:                              │
│                                                                              │
│ /tmp/.local/jumpstarter/venv/lib64/python3.12/site-packages/anyio/from_threa │
│ d.py:131 in run_async_cm                                                     │
│                                                                              │
│   128 │   async def run_async_cm(self) -> bool | None:                       │
│   129 │   │   try:                                                           │
│   130 │   │   │   self._exit_event = Event()                                 │
│ ❱ 131 │   │   │   value = await self._async_cm.__aenter__()                  │
│   132 │   │   except BaseException as exc:                                   │
│   133 │   │   │   self._enter_future.set_exception(exc)                      │
│   134 │   │   │   raise                                                      │
│                                                                              │
│ /usr/lib64/python3.12/contextlib.py:210 in __aenter__                        │
│                                                                              │
│   207 │   │   # they are only needed for recreation, which is not possible a │
│   208 │   │   del self.args, self.kwds, self.func                            │
│   209 │   │   try:                                                           │
│ ❱ 210 │   │   │   return await anext(self.gen)                               │
│   211 │   │   except StopAsyncIteration:                                     │
│   212 │   │   │   raise RuntimeError("generator didn't yield") from None     │
│   213                                                                        │
│                                                                              │
│ /tmp/.local/jumpstarter/venv/lib64/python3.12/site-packages/jumpstarter/conf │
│ ig/client.py:249 in lease_async                                              │
│                                                                              │
│   246 │   │   # when no lease name is provided, release the lease on exit    │
│   247 │   │   release_lease = lease_name == ""                               │
│   248 │   │   try:                                                           │
│ ❱ 249 │   │   │   async with Lease(                                          │
│   250 │   │   │   │   channel=await self.channel(),                          │
│   251 │   │   │   │   namespace=self.metadata.namespace,                     │
│   252 │   │   │   │   name=lease_name,                                       │
│                                                                              │
│ /tmp/.local/jumpstarter/venv/lib64/python3.12/site-packages/anyio/_core/_con │
│ textmanagers.py:163 in __aenter__                                            │
│                                                                              │
│   160 │   │   │   │   f"'yield' statement?"                                  │
│   161 │   │   │   )                                                          │
│   162 │   │                                                                  │
│ ❱ 163 │   │   value = await cm.__aenter__()                                  │
│   164 │   │   self.__cm = cm                                                 │
│   165 │   │   return value                                                   │
│   166                                                                        │
│                                                                              │
│ /usr/lib64/python3.12/contextlib.py:210 in __aenter__                        │
│                                                                              │
│   207 │   │   # they are only needed for recreation, which is not possible a │
│   208 │   │   del self.args, self.kwds, self.func                            │
│   209 │   │   try:                                                           │
│ ❱ 210 │   │   │   return await anext(self.gen)                               │
│   211 │   │   except StopAsyncIteration:                                     │
│   212 │   │   │   raise RuntimeError("generator didn't yield") from None     │
│   213                                                                        │
│                                                                              │
│ /tmp/.local/jumpstarter/venv/lib64/python3.12/site-packages/jumpstarter/clie │
│ nt/lease.py:141 in __asynccontextmanager__                                   │
│                                                                              │
│   138 │                                                                      │
│   139 │   @asynccontextmanager                                               │
│   140 │   async def __asynccontextmanager__(self) -> AsyncGenerator[Self]:   │
│ ❱ 141 │   │   value = await self.request_async()                             │
│   142 │   │   try:                                                           │
│   143 │   │   │   yield value                                                │
│   144 │   │   finally:                                                       │
│                                                                              │
│ /tmp/.local/jumpstarter/venv/lib64/python3.12/site-packages/jumpstarter/clie │
│ nt/lease.py:102 in request_async                                             │
│                                                                              │
│    99 │   │   │   │   await self._create()                                   │
│   100 │   │   else:                                                          │
│   101 │   │   │   await self._create()                                       │
│ ❱ 102 │   │   return await self._acquire()                                   │
│   103 │                                                                      │
│   104 │   async def _acquire(self):                                          │
│   105 │   │   """Acquire a lease.                                            │
│                                                                              │
│ /tmp/.local/jumpstarter/venv/lib64/python3.12/site-packages/jumpstarter/clie │
│ nt/lease.py:109 in _acquire                                                  │
│                                                                              │
│   106 │   │                                                                  │
│   107 │   │   Makes sure the lease is ready, and returns the lease object.   │
│   108 │   │   """                                                            │
│ ❱ 109 │   │   with fail_after(300):  # TODO: configurable timeout            │
│   110 │   │   │   while True:                                                │
│   111 │   │   │   │   logger.debug("Polling Lease %s", self.name)            │
│   112 │   │   │   │   result = await self.get()                              │
│                                                                              │
│ /usr/lib64/python3.12/contextlib.py:158 in __exit__                          │
│                                                                              │
│   155 │   │   │   │   # tell if we get the same exception back               │
│   156 │   │   │   │   value = typ()                                          │
│   157 │   │   │   try:                                                       │
│ ❱ 158 │   │   │   │   self.gen.throw(value)                                  │
│   159 │   │   │   except StopIteration as exc:                               │
│   160 │   │   │   │   # Suppress StopIteration *unless* it's the same except │
│   161 │   │   │   │   # was passed to throw().  This prevents a StopIteratio │
│                                                                              │
│ /tmp/.local/jumpstarter/venv/lib64/python3.12/site-packages/anyio/_core/_tas │
│ ks.py:123 in fail_after                                                      │
│                                                                              │
│   120 │   │   yield cancel_scope                                             │
│   121 │                                                                      │
│   122 │   if cancel_scope.cancelled_caught and current_time() >= cancel_scop │
│ ❱ 123 │   │   raise TimeoutError                                             │
│   124                                                                        │
│   125                                                                        │
│   126 def move_on_after(delay: float | None, shield: bool = False) -> Cancel │
╰──────────────────────────────────────────────────────────────────────────────╯

[mangelajo]

Leasing directly from pytest, it doesn't even seem to wait for a lease:

+ sudo podman cp 1c9608077991447a52ecd46d61b602c1eb9926e2ad6561b42c48822eceab403d:/disk.raw.xz output/image/disk.raw.xz
+ sudo podman rm 1c9608077991447a52ecd46d61b602c1eb9926e2ad6561b42c48822eceab403d
1c9608077991447a52ecd46d61b602c1eb9926e2ad6561b42c48822eceab403d
++ du -h output/image/disk.raw.xz
++ cut -f1
+ printf 'Disk image extracted: %s\n' 1.7G
Disk image extracted: 1.7G
+ echo 'Running the Python jumpstarter test'
+ tee jumpstarter.log
Running the Python jumpstarter test
+ pytest -s scripts/testing/jumpstarter-scripts/orin-kmods.py
+ tee -a jumpstarter.log
============================= test session starts ==============================
platform linux -- Python 3.12.11, pytest-8.4.2, pluggy-1.6.0
rootdir: /builds/redhat/rhel/sst/orin-sidecar/nvidia-jetson-sidecar
plugins: anyio-4.11.0, cov-7.0.0, asyncio-1.2.0
asyncio: mode=Mode.STRICT, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function
collected 5 items
scripts/testing/jumpstarter-scripts/orin-kmods.py EEEEE
==================================== ERRORS ====================================
________________ ERROR at setup of TestOrin.test_nvidia_devices ________________
fixturedef = <FixtureDef argname='client' scope='class' baseid='scripts/testing/jumpstarter-scripts/orin-kmods.py::TestOrin'>
request = <SubRequest 'client' for <Function test_nvidia_devices>>
    @pytest.hookimpl(wrapper=True)
    def pytest_fixture_setup(fixturedef: FixtureDef, request) -> object | None:
        asyncio_mode = _get_asyncio_mode(request.config)
        if not _is_asyncio_fixture_function(fixturedef.func):
            if asyncio_mode == Mode.STRICT:
                # Ignore async fixtures without explicit asyncio mark in strict mode
                # This applies to pytest_trio fixtures, for example
>               return (yield)
                        ^^^^^
/tmp/.local/jumpstarter/venv/lib64/python3.12/site-packages/pytest_asyncio/plugin.py:[733](https://gitlab.com/redhat/rhel/sst/orin-sidecar/nvidia-jetson-sidecar/-/jobs/11532231849#L733): 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/tmp/.local/jumpstarter/venv/lib64/python3.12/site-packages/jumpstarter_testing/pytest.py:56: in client
    with env() as client:
         ^^^^^
/usr/lib64/python3.12/contextlib.py:137: in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
/tmp/.local/jumpstarter/venv/lib64/python3.12/site-packages/jumpstarter/utils/env.py:49: in env
    with portal.wrap_async_context_manager(env_async(portal, stack)) as client:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/tmp/.local/jumpstarter/venv/lib64/python3.12/site-packages/anyio/from_thread.py:155: in __enter__
    return self._enter_future.result()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
/usr/lib64/python3.12/concurrent/futures/_base.py:456: in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
/usr/lib64/python3.12/concurrent/futures/_base.py:401: in __get_result
    raise self._exception
/tmp/.local/jumpstarter/venv/lib64/python3.12/site-packages/anyio/from_thread.py:252: in _call_func
    retval = await retval_or_awaitable
             ^^^^^^^^^^^^^^^^^^^^^^^^^
/tmp/.local/jumpstarter/venv/lib64/python3.12/site-packages/anyio/from_thread.py:131: in run_async_cm
    value = await self._async_cm.__aenter__()
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/usr/lib64/python3.12/contextlib.py:210: in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
/tmp/.local/jumpstarter/venv/lib64/python3.12/site-packages/jumpstarter/utils/env.py:26: in env_async
    async with client_from_path(
/usr/lib64/python3.12/contextlib.py:210: in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
/tmp/.local/jumpstarter/venv/lib64/python3.12/site-packages/jumpstarter/client/client.py:20: in client_from_path
    yield await client_from_channel(channel, portal, stack, allow, unsafe)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/tmp/.local/jumpstarter/venv/lib64/python3.12/site-packages/jumpstarter/client/client.py:37: in client_from_channel
    response = await stub.GetReport(empty_pb2.Empty())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
self = <_AioCall of RPC that terminated with:
	status = failed to connect to all addresses; last error: UNKNOWN: unix:/tmp/ju...rror: UNKNOWN: unix:/tmp/jumpstarter-g7espbpn/socket: connection attempt timed out before receiving SETTINGS frame"}"
>
    def __await__(self) -> Generator[Any, None, ResponseType]:
        """Wait till the ongoing RPC request finishes."""
        try:
            response = yield from self._call_response
        except asyncio.CancelledError:
            # Even if we caught all other CancelledError, there is still
            # this corner case. If the application cancels immediately after
            # the Call object is created, we will observe this
            # `CancelledError`.
            if not self.cancelled():
                self.cancel()
            raise
    
        # NOTE(lidiz) If we raise RpcError in the task, and users doesn't
        # 'await' on it. AsyncIO will log 'Task exception was never retrieved'.
        # Instead, if we move the exception raising here, the spam stops.
        # Unfortunately, there can only be one 'yield from' in '__await__'. So,
        # we need to access the private instance variable.
        if response is cygrpc.EOF:
            if self._cython_call.is_locally_cancelled():
                raise asyncio.CancelledError()
            else:
>               raise _create_rpc_error(
                    self._cython_call._initial_metadata,
                    self._cython_call._status,
                )
E               grpc.aio._call.AioRpcError: <AioRpcError of RPC that terminated with:
E               	status = StatusCode.UNAVAILABLE
E               	details = "failed to connect to all addresses; last error: UNKNOWN: unix:/tmp/jumpstarter-g7espbpn/socket: connection attempt timed out before receiving SETTINGS frame"
E               	debug_error_string = "UNKNOWN:Error received from peer  {grpc_status:14, grpc_message:"failed to connect to all addresses; last error: UNKNOWN: unix:/tmp/jumpstarter-g7espbpn/socket: connection attempt timed out before receiving SETTINGS frame"}"
E               >
/tmp/.local/jumpstarter/venv/lib64/python3.12/site-packages/grpc/aio/_call.py:330: AioRpcError
________________ ERROR at setup of TestOrin.test_tegra_devices _________________
self = <orin-kmods.TestOrin object at 0xffff8cbb37a0>
    @pytest.fixture(scope="class")
    def client(self):
        try:
>           with env() as client:
                 ^^^^^
/tmp/.local/jumpstarter/venv/lib64/python3.12/site-packages/jumpstarter_testing/pytest.py:56: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/usr/lib64/python3.12/contextlib.py:137: in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
/tmp/.local/jumpstarter/venv/lib64/python3.12/site-packages/jumpstarter/utils/env.py:49: in env
    with portal.wrap_async_context_manager(env_async(portal, stack)) as client:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/tmp/.local/jumpstarter/venv/lib64/python3.12/site-packages/anyio/from_thread.py:155: in __enter__
    return self._enter_future.result()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
/usr/lib64/python3.12/concurrent/futures/_base.py:456: in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
/usr/lib64/python3.12/concurrent/futures/_base.py:401: in __get_result
    raise self._exception
/tmp/.local/jumpstarter/venv/lib64/python3.12/site-packages/anyio/from_thread.py:252: in _call_func
    retval = await retval_or_awaitable
             ^^^^^^^^^^^^^^^^^^^^^^^^^
/tmp/.local/jumpstarter/venv/lib64/python3.12/site-packages/anyio/from_thread.py:131: in run_async_cm
    value = await self._async_cm.__aenter__()
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/usr/lib64/python3.12/contextlib.py:210: in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
/tmp/.local/jumpstarter/venv/lib64/python3.12/site-packages/jumpstarter/utils/env.py:26: in env_async
    async with client_from_path(
/usr/lib64/python3.12/contextlib.py:210: in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
/tmp/.local/jumpstarter/venv/lib64/python3.12/site-packages/jumpstarter/client/client.py:20: in client_from_path
    yield await client_from_channel(channel, portal, stack, allow, unsafe)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/tmp/.local/jumpstarter/venv/lib64/python3.12/site-packages/jumpstarter/client/client.py:37: in client_from_channel
    response = await stub.GetReport(empty_pb2.Empty())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
self = <_AioCall of RPC that terminated with:
	status = failed to connect to all addresses; last error: UNKNOWN: unix:/tmp/ju...rror: UNKNOWN: unix:/tmp/jumpstarter-g7espbpn/socket: connection attempt timed out before receiving SETTINGS frame"}"
>
    def __await__(self) -> Generator[Any, None, ResponseType]:
        """Wait till the ongoing RPC request finishes."""
        try:
            response = yield from self._call_response
        except asyncio.CancelledError:
            # Even if we caught all other CancelledError, there is still
            # this corner case. If the application cancels immediately after
            # the Call object is created, we will observe this
            # `CancelledError`.
            if not self.cancelled():
                self.cancel()
            raise
    
        # NOTE(lidiz) If we raise RpcError in the task, and users doesn't
        # 'await' on it. AsyncIO will log 'Task exception was never retrieved'.
        # Instead, if we move the exception raising here, the spam stops.
        # Unfortunately, there can only be one 'yield from' in '__await__'. So,
        # we need to access the private instance variable.
        if response is cygrpc.EOF:
            if self._cython_call.is_locally_cancelled():
                raise asyncio.CancelledError()
            else:
>               raise _create_rpc_error(
                    self._cython_call._initial_metadata,
                    self._cython_call._status,
                )
E               grpc.aio._call.AioRpcError: <AioRpcError of RPC that terminated with:
E               	status = StatusCode.UNAVAILABLE
E               	details = "failed to connect to all addresses; last error: UNKNOWN: unix:/tmp/jumpstarter-g7espbpn/socket: connection attempt timed out before receiving SETTINGS frame"
E               	debug_error_string = "UNKNOWN:Error received from peer  {grpc_status:14, grpc_message:"failed to connect to all addresses; last error: UNKNOWN: unix:/tmp/jumpstarter-g7espbpn/socket: connection attempt timed out before receiving SETTINGS frame"}"
E               >
/tmp/.local/jumpstarter/venv/lib64/python3.12/site-packages/grpc/aio/_call.py:330: AioRpcError
____________ ERROR at setup of TestOrin.test_nvidia_kernel_modules _____________
self = <orin-kmods.TestOrin object at 0xffff8cbb37a0>
    @pytest.fixture(scope="class")
    def client(self):
        try:
>           with env() as client:
                 ^^^^^
/tmp/.local/jumpstarter/venv/lib64/python3.12/site-packages/jumpstarter_testing/pytest.py:56: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/usr/lib64/python3.12/contextlib.py:137: in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
/tmp/.local/jumpstarter/venv/lib64/python3.12/site-packages/jumpstarter/utils/env.py:49: in env
    with portal.wrap_async_context_manager(env_async(portal, stack)) as client:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/tmp/.local/jumpstarter/venv/lib64/python3.12/site-packages/anyio/from_thread.py:155: in __enter__
    return self._enter_future.result()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
/usr/lib64/python3.12/concurrent/futures/_base.py:456: in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
/usr/lib64/python3.12/concurrent/futures/_base.py:401: in __get_result
    raise self._exception
/tmp/.local/jumpstarter/venv/lib64/python3.12/site-packages/anyio/from_thread.py:252: in _call_func
    retval = await retval_or_awaitable
             ^^^^^^^^^^^^^^^^^^^^^^^^^
/tmp/.local/jumpstarter/venv/lib64/python3.12/site-packages/anyio/from_thread.py:131: in run_async_cm
    value = await self._async_cm.__aenter__()
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/usr/lib64/python3.12/contextlib.py:210: in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
/tmp/.local/jumpstarter/venv/lib64/python3.12/site-packages/jumpstarter/utils/env.py:26: in env_async
    async with client_from_path(
/usr/lib64/python3.12/contextlib.py:210: in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
/tmp/.local/jumpstarter/venv/lib64/python3.12/site-packages/jumpstarter/client/client.py:20: in client_from_path
    yield await client_from_channel(channel, portal, stack, allow, unsafe)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/tmp/.local/jumpstarter/venv/lib64/python3.12/site-packages/jumpstarter/client/client.py:37: in client_from_channel
    response = await stub.GetReport(empty_pb2.Empty())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
self = <_AioCall of RPC that terminated with:
	status = failed to connect to all addresses; last error: UNKNOWN: unix:/tmp/ju...rror: UNKNOWN: unix:/tmp/jumpstarter-g7espbpn/socket: connection attempt timed out before receiving SETTINGS frame"}"
>
    def __await__(self) -> Generator[Any, None, ResponseType]:
        """Wait till the ongoing RPC request finishes."""
        try:
            response = yield from self._call_response
        except asyncio.CancelledError:
            # Even if we caught all other CancelledError, there is still
            # this corner case. If the application cancels immediately after
            # the Call object is created, we will observe this
            # `CancelledError`.
            if not self.cancelled():
                self.cancel()
            raise
    
        # NOTE(lidiz) If we raise RpcError in the task, and users doesn't
        # 'await' on it. AsyncIO will log 'Task exception was never retrieved'.
        # Instead, if we move the exception raising here, the spam stops.
        # Unfortunately, there can only be one 'yield from' in '__await__'. So,
        # we need to access the private instance variable.
        if response is cygrpc.EOF:
            if self._cython_call.is_locally_cancelled():
                raise asyncio.CancelledError()
            else:
>               raise _create_rpc_error(
                    self._cython_call._initial_metadata,
                    self._cython_call._status,
                )
E               grpc.aio._call.AioRpcError: <AioRpcError of RPC that terminated with:
E               	status = StatusCode.UNAVAILABLE
E               	details = "failed to connect to all addresses; last error: UNKNOWN: unix:/tmp/jumpstarter-g7espbpn/socket: connection attempt timed out before receiving SETTINGS frame"
E               	debug_error_string = "UNKNOWN:Error received from peer  {grpc_status:14, grpc_message:"failed to connect to all addresses; last error: UNKNOWN: unix:/tmp/jumpstarter-g7espbpn/socket: connection attempt timed out before receiving SETTINGS frame"}"
E               >
/tmp/.local/jumpstarter/venv/lib64/python3.12/site-packages/grpc/aio/_call.py:330: AioRpcError
______________ ERROR at setup of TestOrin.test_cuda_device_query _______________
self = <orin-kmods.TestOrin object at 0xffff8cbb37a0>
    @pytest.fixture(scope="class")
    def client(self):
        try:
>           with env() as client:
                 ^^^^^
/tmp/.local/jumpstarter/venv/lib64/python3.12/site-packages/jumpstarter_testing/pytest.py:56: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/usr/lib64/python3.12/contextlib.py:137: in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
/tmp/.local/jumpstarter/venv/lib64/python3.12/site-packages/jumpstarter/utils/env.py:49: in env
    with portal.wrap_async_context_manager(env_async(portal, stack)) as client:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/tmp/.local/jumpstarter/venv/lib64/python3.12/site-packages/anyio/from_thread.py:155: in __enter__
    return self._enter_future.result()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
/usr/lib64/python3.12/concurrent/futures/_base.py:456: in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
/usr/lib64/python3.12/concurrent/futures/_base.py:401: in __get_result
    raise self._exception
/tmp/.local/jumpstarter/venv/lib64/python3.12/site-packages/anyio/from_thread.py:252: in _call_func
    retval = await retval_or_awaitable
             ^^^^^^^^^^^^^^^^^^^^^^^^^
/tmp/.local/jumpstarter/venv/lib64/python3.12/site-packages/anyio/from_thread.py:131: in run_async_cm
    value = await self._async_cm.__aenter__()
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/usr/lib64/python3.12/contextlib.py:210: in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
/tmp/.local/jumpstarter/venv/lib64/python3.12/site-packages/jumpstarter/utils/env.py:26: in env_async
    async with client_from_path(
/usr/lib64/python3.12/contextlib.py:210: in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
/tmp/.local/jumpstarter/venv/lib64/python3.12/site-packages/jumpstarter/client/client.py:20: in client_from_path
    yield await client_from_channel(channel, portal, stack, allow, unsafe)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/tmp/.local/jumpstarter/venv/lib64/python3.12/site-packages/jumpstarter/client/client.py:37: in client_from_channel
    response = await stub.GetReport(empty_pb2.Empty())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
self = <_AioCall of RPC that terminated with:
	status = failed to connect to all addresses; last error: UNKNOWN: unix:/tmp/ju...rror: UNKNOWN: unix:/tmp/jumpstarter-g7espbpn/socket: connection attempt timed out before receiving SETTINGS frame"}"
>
    def __await__(self) -> Generator[Any, None, ResponseType]:
        """Wait till the ongoing RPC request finishes."""
        try:
            response = yield from self._call_response
        except asyncio.CancelledError:
            # Even if we caught all other CancelledError, there is still
            # this corner case. If the application cancels immediately after
            # the Call object is created, we will observe this
            # `CancelledError`.
            if not self.cancelled():
                self.cancel()
            raise
    
        # NOTE(lidiz) If we raise RpcError in the task, and users doesn't
        # 'await' on it. AsyncIO will log 'Task exception was never retrieved'.
        # Instead, if we move the exception raising here, the spam stops.
        # Unfortunately, there can only be one 'yield from' in '__await__'. So,
        # we need to access the private instance variable.
        if response is cygrpc.EOF:
            if self._cython_call.is_locally_cancelled():
                raise asyncio.CancelledError()
            else:
>               raise _create_rpc_error(
                    self._cython_call._initial_metadata,
                    self._cython_call._status,
                )
E               grpc.aio._call.AioRpcError: <AioRpcError of RPC that terminated with:
E               	status = StatusCode.UNAVAILABLE
E               	details = "failed to connect to all addresses; last error: UNKNOWN: unix:/tmp/jumpstarter-g7espbpn/socket: connection attempt timed out before receiving SETTINGS frame"
E               	debug_error_string = "UNKNOWN:Error received from peer  {grpc_status:14, grpc_message:"failed to connect to all addresses; last error: UNKNOWN: unix:/tmp/jumpstarter-g7espbpn/socket: connection attempt timed out before receiving SETTINGS frame"}"
E               >
/tmp/.local/jumpstarter/venv/lib64/python3.12/site-packages/grpc/aio/_call.py:330: AioRpcError
_____________ ERROR at setup of TestOrin.test_cuda_bandwidth_test ______________
self = <orin-kmods.TestOrin object at 0xffff8cbb37a0>
    @pytest.fixture(scope="class")
    def client(self):
        try:
>           with env() as client:
                 ^^^^^
/tmp/.local/jumpstarter/venv/lib64/python3.12/site-packages/jumpstarter_testing/pytest.py:56: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/usr/lib64/python3.12/contextlib.py:137: in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
/tmp/.local/jumpstarter/venv/lib64/python3.12/site-packages/jumpstarter/utils/env.py:49: in env
    with portal.wrap_async_context_manager(env_async(portal, stack)) as client:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/tmp/.local/jumpstarter/venv/lib64/python3.12/site-packages/anyio/from_thread.py:155: in __enter__
    return self._enter_future.result()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
/usr/lib64/python3.12/concurrent/futures/_base.py:456: in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
/usr/lib64/python3.12/concurrent/futures/_base.py:401: in __get_result
    raise self._exception
/tmp/.local/jumpstarter/venv/lib64/python3.12/site-packages/anyio/from_thread.py:252: in _call_func
    retval = await retval_or_awaitable
             ^^^^^^^^^^^^^^^^^^^^^^^^^
/tmp/.local/jumpstarter/venv/lib64/python3.12/site-packages/anyio/from_thread.py:131: in run_async_cm
    value = await self._async_cm.__aenter__()
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/usr/lib64/python3.12/contextlib.py:210: in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
/tmp/.local/jumpstarter/venv/lib64/python3.12/site-packages/jumpstarter/utils/env.py:26: in env_async
    async with client_from_path(
/usr/lib64/python3.12/contextlib.py:210: in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
/tmp/.local/jumpstarter/venv/lib64/python3.12/site-packages/jumpstarter/client/client.py:20: in client_from_path
    yield await client_from_channel(channel, portal, stack, allow, unsafe)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/tmp/.local/jumpstarter/venv/lib64/python3.12/site-packages/jumpstarter/client/client.py:37: in client_from_channel
    response = await stub.GetReport(empty_pb2.Empty())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
self = <_AioCall of RPC that terminated with:
	status = failed to connect to all addresses; last error: UNKNOWN: unix:/tmp/ju...rror: UNKNOWN: unix:/tmp/jumpstarter-g7espbpn/socket: connection attempt timed out before receiving SETTINGS frame"}"
>
    def __await__(self) -> Generator[Any, None, ResponseType]:
        """Wait till the ongoing RPC request finishes."""
        try:
            response = yield from self._call_response
        except asyncio.CancelledError:
            # Even if we caught all other CancelledError, there is still
            # this corner case. If the application cancels immediately after
            # the Call object is created, we will observe this
            # `CancelledError`.
            if not self.cancelled():
                self.cancel()
            raise
    
        # NOTE(lidiz) If we raise RpcError in the task, and users doesn't
        # 'await' on it. AsyncIO will log 'Task exception was never retrieved'.
        # Instead, if we move the exception raising here, the spam stops.
        # Unfortunately, there can only be one 'yield from' in '__await__'. So,
        # we need to access the private instance variable.
        if response is cygrpc.EOF:
            if self._cython_call.is_locally_cancelled():
                raise asyncio.CancelledError()
            else:
>               raise _create_rpc_error(
                    self._cython_call._initial_metadata,
                    self._cython_call._status,
                )
E               grpc.aio._call.AioRpcError: <AioRpcError of RPC that terminated with:
E               	status = StatusCode.UNAVAILABLE
E               	details = "failed to connect to all addresses; last error: UNKNOWN: unix:/tmp/jumpstarter-g7espbpn/socket: connection attempt timed out before receiving SETTINGS frame"
E               	debug_error_string = "UNKNOWN:Error received from peer  {grpc_status:14, grpc_message:"failed to connect to all addresses; last error: UNKNOWN: unix:/tmp/jumpstarter-g7espbpn/socket: connection attempt timed out before receiving SETTINGS frame"}"
E               >
/tmp/.local/jumpstarter/venv/lib64/python3.12/site-packages/grpc/aio/_call.py:330: AioRpcError
=========================== short test summary info ============================
ERROR scripts/testing/jumpstarter-scripts/orin-kmods.py::TestOrin::test_nvidia_devices - grpc.aio._call.AioRpcError: <AioRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "failed to connect to all addresses; last error: UNKNOWN: unix:/tmp/jumpstarter-g7espbpn/socket: connection attempt timed out before receiving SETTINGS frame"
	debug_error_string = "UNKNOWN:Error received from peer  {grpc_status:14, grpc_message:"failed to connect to all addresses; last error: UNKNOWN: unix:/tmp/jumpstarter-g7espbpn/socket: connection attempt timed out before receiving SETTINGS frame"}"
>
ERROR scripts/testing/jumpstarter-scripts/orin-kmods.py::TestOrin::test_tegra_devices - grpc.aio._call.AioRpcError: <AioRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "failed to connect to all addresses; last error: UNKNOWN: unix:/tmp/jumpstarter-g7espbpn/socket: connection attempt timed out before receiving SETTINGS frame"
	debug_error_string = "UNKNOWN:Error received from peer  {grpc_status:14, grpc_message:"failed to connect to all addresses; last error: UNKNOWN: unix:/tmp/jumpstarter-g7espbpn/socket: connection attempt timed out before receiving SETTINGS frame"}"
>
ERROR scripts/testing/jumpstarter-scripts/orin-kmods.py::TestOrin::test_nvidia_kernel_modules - grpc.aio._call.AioRpcError: <AioRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "failed to connect to all addresses; last error: UNKNOWN: unix:/tmp/jumpstarter-g7espbpn/socket: connection attempt timed out before receiving SETTINGS frame"
	debug_error_string = "UNKNOWN:Error received from peer  {grpc_status:14, grpc_message:"failed to connect to all addresses; last error: UNKNOWN: unix:/tmp/jumpstarter-g7espbpn/socket: connection attempt timed out before receiving SETTINGS frame"}"
>
ERROR scripts/testing/jumpstarter-scripts/orin-kmods.py::TestOrin::test_cuda_device_query - grpc.aio._call.AioRpcError: <AioRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "failed to connect to all addresses; last error: UNKNOWN: unix:/tmp/jumpstarter-g7espbpn/socket: connection attempt timed out before receiving SETTINGS frame"
	debug_error_string = "UNKNOWN:Error received from peer  {grpc_status:14, grpc_message:"failed to connect to all addresses; last error: UNKNOWN: unix:/tmp/jumpstarter-g7espbpn/socket: connection attempt timed out before receiving SETTINGS frame"}"
>
ERROR scripts/testing/jumpstarter-scripts/orin-kmods.py::TestOrin::test_cuda_bandwidth_test - grpc.aio._call.AioRpcError: <AioRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "failed to connect to all addresses; last error: UNKNOWN: unix:/tmp/jumpstarter-g7espbpn/socket: connection attempt timed out before receiving SETTINGS frame"
	debug_error_string = "UNKNOWN:Error received from peer  {grpc_status:14, grpc_message:"failed to connect to all addresses; last error: UNKNOWN: unix:/tmp/jumpstarter-g7espbpn/socket: connection attempt timed out before receiving SETTINGS frame"}"
>
============================== 5 errors in 20.97s ==============================
+ cleanup
+ printf 'Fixing the permissions on the bib artifacts ...\n'
Fixing the permissions on the bib artifacts ...
++ id -u
++ id -g
+ sudo chown -Rv 0:0 output
ownership of 'output/image/disk.raw.xz' retained as root:root
ownership of 'output/image' retained as root:root
ownership of 'output' retained as root:root
[09/29/25 11:25:10] INFO     INFO:jumpstarter.client.lease:Releasin lease.py:146
                             g Lease                                            
                             0199[953](https://gitlab.com/redhat/rhel/sst/orin-sidecar/nvidia-jetson-sidecar/-/jobs/11532231849#L953)6-7b75-7b3c-8e07-8fdff16fae43               
Running after_script
00:03
Running after script...
$ sudo rm -rf output
$ sudo podman rmi --force --ignore ${BOOTC_TEST_IMAGE} ${BOOTC_IMAGE_BUILDER}
$ sudo podman image prune --force