[Question] how should it be used in the API?

[ltalirz]

@jan-janssen Here is a mockup of how I was planning to use executorlib in the amorphouspy API [1]

Note that this currently fails - f2 is never done()

"""Reproduce submit/check pattern: submit with one executor, check with another."""

import tempfile
import time
from pathlib import Path

# doesn't matter whether we use TestClusterExecutor or SingleNodeExecutor
# from executorlib.api import TestClusterExecutor as MyExecutor  
from executorlib import SingleNodeExecutor as MyExecutor


def slow_add(a, b):
    time.sleep(4)
    return a + b


if __name__ == "__main__":
    with tempfile.TemporaryDirectory() as tmpdir:
        cache_dir = Path(tmpdir) / "jobs"

        # Submit
        exe = MyExecutor(cache_directory=cache_dir)
        future = exe.submit(slow_add, 2, 3)
        exe.shutdown(wait=False, cancel_futures=False)
        print(f"submitted, done={future.done()}")

        # Poll with new executor every 2s
        for i in range(10):
            time.sleep(2)
            exe2 = MyExecutor(cache_directory=cache_dir)
            f2 = exe2.submit(slow_add, 2, 3)
            exe2.shutdown(wait=False, cancel_futures=False)
            print(f"poll {i+1}: done={f2.done()}")
            if f2.done():
                print(f"result={f2.result()}")
                break
        else:
            print("TIMEOUT")

[1] This is a slight simplification since in the API the submit and poll stages run in separate threads.

[ltalirz]

To be clear, I would actually prefer a more direct way of checking job completion over running the submission again - if you can provide the suggested route, I will take it (ideally I should be able to store an identifier and then load the right file directly without having to go through a list of all that are available).

[jan-janssen]

With tempfile.TemporaryDirectory() you always create a temporary directory which is deleted after the context is closed:
https://docs.python.org/3/library/tempfile.html#tempfile.TemporaryDirectory

This class securely creates a temporary directory using the same rules as mkdtemp(). The resulting object can be used as a context manager (see Examples). On completion of the context or destruction of the temporary directory object, the newly created temporary directory and all its contents are removed from the filesystem.

So this cannot work.

In terms of the status checking, this is now implemented in #915 as well as the deletion of future objects #914 .

[ltalirz]

With tempfile.TemporaryDirectory() you always create a temporary directory which is deleted after the context is closed: https://docs.python.org/3/library/tempfile.html#tempfile.TemporaryDirectory

This class securely creates a temporary directory using the same rules as mkdtemp(). The resulting object can be used as a context manager (see Examples). On completion of the context or destruction of the temporary directory object, the newly created temporary directory and all its contents are removed from the filesystem.

So this cannot work.

The tempdir is at the top level of the script. It is only deleted after the the loop is exited or timed out. Both executors use the same cache dir

The tempdir is not necessary to demonstrate this behavior, it just cleans up files after the script exits.

[ltalirz]

In terms of the status checking, this is now implemented in #915 as well as the deletion of future objects #914 .

Great, let me have a look.

[ltalirz]

Ok, yes I can set and store the cache_key myself (if you prefer that over using some unique identifier that is already assigned by executorlib) and then use it to retrieve the future in /check.

[jan-janssen]

The interface for the SingleNodeExecutor should be fixed in #911 - this is included in executorlib-1.8.1-pyhcf101f3_2. I am going to release a 1.8.2 version tonight then we can test with that one.

[jan-janssen]

Ok, yes I can set and store the cache_key myself (if you prefer that over using some unique identifier that is already assigned by executorlib) and then use it to retrieve the future in /check.

Yes, I think this is easier for frameworks which want to use executorlib as backend - I plan to add a corresponding section to the documentation, at the moment I collect the notes in #916

[jan-janssen]

I am going to move this issue to the executorlib repository, that helps me to keep track of the issues.

[jan-janssen]

@ltalirz I transferred the issue to the executorlib repository.

[ltalirz]

Btw, when I modify this to use get_future_from_cache, I get the FileNotFoundError that I also see on CI for the integration test in the amorphouspy PR

"""Reproduce submit/check pattern: submit once, save the cache key, then
use get_future_from_cache() to recreate the future for polling.

See https://github.com/pyiron/executorlib/pull/915
"""

import tempfile
import time
from pathlib import Path

# doesn't matter whether we use TestClusterExecutor or SingleNodeExecutor
# from executorlib.api import TestClusterExecutor as MyExecutor
from executorlib import SingleNodeExecutor as MyExecutor
from executorlib import get_future_from_cache


def slow_add(a, b):
    time.sleep(4)
    return a + b


if __name__ == "__main__":
    with tempfile.TemporaryDirectory() as tmpdir:
        cache_dir = Path(tmpdir) / "jobs"
        cache_key = "slow_add_task"

        # Submit with an explicit cache_key so we can look it up later
        exe = MyExecutor(cache_directory=cache_dir)
        future = exe.submit(
            slow_add,
            2,
            3,
            resource_dict={"cache_key": cache_key},
        )
        exe.shutdown(wait=False, cancel_futures=False)
        print(f"submitted, done={future.done()}")

        # Poll by recreating the future from the cache every 2s
        for i in range(10):
            time.sleep(2)
            f2 = get_future_from_cache(
                cache_directory=str(cache_dir),
                cache_key=cache_key,
            )
            print(f"poll {i+1}: done={f2.done()}")
            if f2.done():
                print(f"result={f2.result()}")
                break
        else:
            print("TIMEOUT")

 python test_executorlib.py 
submitted, done=False
Traceback (most recent call last):
  File "/Users/leopold/Personal/Postdoc-MARVEL/repos/amorphouspy/amorphouspy_api/test_executorlib.py", line 41, in <module>
    f2 = get_future_from_cache(
        cache_directory=str(cache_dir),
        cache_key=cache_key,
    )
  File "/Users/leopold/Applications/miniforge3/envs/amorphouspy/lib/python3.13/site-packages/executorlib/__init__.py", line 67, in get_future_from_cache
    return get_future_from_cache(
        cache_directory=cache_directory,
        cache_key=cache_key,
    )
  File "/Users/leopold/Applications/miniforge3/envs/amorphouspy/lib/python3.13/site-packages/executorlib/standalone/hdf.py", line 191, in get_future_from_cache
    raise FileNotFoundError(
        f"Neither input nor output file for cache key {cache_key} found in cache directory {cache_directory}."
    )
FileNotFoundError: Neither input nor output file for cache key slow_add_task found in cache directory /var/folders/8y/7c7qkxcx5vxb749rt3sw5sh40000gp/T/tmp0v3if955/jobs.

[ltalirz]

That's for the SingleNodeExecutor - with the TestClusterExecutor it works