[Documentation] Debugging file based executors

[jan-janssen]

The SlurmClusterExecutor and FluxClusterExecutor behave fundamentally different from the SingleNodeExecutor as explained in the documentation: https://executorlib.readthedocs.io/en/latest/4-developer.html#interface-class-hierarchy

This makes it hard to locally debug their functionality. Still internally executorlib contains the functionality to build a SingleNodeExecutor which behaves similar to the SlurmClusterExecutor and FluxClusterExecutor. This should be explained in the documentation:

import os
import time
from typing import Optional, Callable
from executorlib.task_scheduler.file.subprocess_spawner import execute_in_subprocess
from executorlib.task_scheduler.file.task_scheduler import FileTaskScheduler
from executorlib.executor.base import BaseExecutor


class LocalFileExecutor(BaseExecutor):
 def __init__(
 self,
 max_workers: Optional[int] = None,
 cache_directory: Optional[str] = None,
 max_cores: Optional[int] = None,
 resource_dict: Optional[dict] = None,
 hostname_localhost: Optional[bool] = None,
 block_allocation: bool = False,
 init_function: Optional[Callable] = None,
 disable_dependencies: bool = False,
 refresh_rate: float = 0.01,
 ):
 default_resource_dict: dict = {
 "cores": 1,
 "threads_per_core": 1,
 "gpus_per_core": 0,
 "cwd": None,
 "openmpi_oversubscribe": False,
 "slurm_cmd_args": [],
 }
 if cache_directory is None:
 default_resource_dict["cache_directory"] = "executorlib_cache"
 else:
 default_resource_dict["cache_directory"] = cache_directory
 if resource_dict is None:
 resource_dict = {}
 resource_dict.update(
 {k: v for k, v in default_resource_dict.items() if k not in resource_dict}
 )
 super().__init__(
 executor=FileTaskScheduler(
 resource_dict=resource_dict,
 pysqa_config_directory=None,
 backend=None,
 disable_dependencies=disable_dependencies,
 execute_function=execute_in_subprocess,
 )
 )


def foo(x):
 time.sleep(3)
 return x + 1


with LocalFileExecutor(
 cache_directory="not_this_dir",
 resource_dict={}
) as exe:
 future = exe.submit(
 foo, 
 1, 
 resource_dict={
 "cache_directory": "rather_this_dir", 
 "cache_key": "foo",
 },
 )
 print(future.result())

[liamhuber]

@jan-janssen, when I run the example I'm afraid neither the not_this_dir nor the rather_this_dir get used, but the default executorlib_cache. Your subclass here nicely passes the not_this_dir value into the FileTaskScheduler resource_dict argument correctly, so it must be getting thrown out somewhere under the hood later.

[liamhuber]

Nevermind! What I said is true on main, but the example works fine on the #707 branch. Once #707 is merged, this example is fine.

[jan-janssen]

While the corresponding Executor was added in #714 it is not yet explained in the documentation.

[jan-janssen]

This functionality was documented in #751