stouputils.parallel.subprocess module#

exception RemoteSubprocessError(
exc_type: str,
exc_repr: str,
traceback_str: str,
)[source]#

Bases: RuntimeError

Raised in the parent when the child raised an exception - contains the child’s formatted traceback.

run_in_subprocess(
func: Callable[[...], R],
*args: Any,
timeout: float | None = None,
no_join: bool = False,
capture_output: bool = True,
process_title: str | None = None,
**kwargs: Any,
) → R[source]#

Execute a function in a subprocess with positional and keyword arguments.

This is useful when you need to run a function in isolation to avoid memory leaks, resource conflicts, or to ensure a clean execution environment. The subprocess will be created, run the function with the provided arguments, and return the result.

Parameters:

  • func – The function to execute in a subprocess. (SHOULD BE A TOP-LEVEL FUNCTION TO BE PICKLABLE)

  • *args – Positional arguments to pass to the function.

  • timeout – Maximum time in seconds to wait for the subprocess. If None, wait indefinitely. If the subprocess exceeds this time, it will be terminated.

  • no_join – If True, do not wait for the subprocess to finish (fire-and-forget) and return the Process object.

  • capture_output – If True, capture the subprocess’ stdout/stderr and relay it in real time to the parent’s stdout. This enables seeing print() output from the subprocess in the main process.

  • process_title – If provided, sets the process title visible in process lists. If it starts with ‘+++’, this prefix is replaced by the current process title.

  • **kwargs – Keyword arguments to pass to the function.

Returns:

The return value of the function.

Raises:

  • Exception – The child’s own exception, when it survives a pickle round trip. It is chained from a RemoteSubprocessError holding the child’s formatted traceback.

  • RemoteSubprocessError – If the child raised an exception that cannot be pickled back.

  • RuntimeError – If the subprocess exits with a non-zero exit code or did not return a result.

  • TimeoutError – If the subprocess exceeds the specified timeout.

import stouputils as stp

def train(epochs: int, learning_rate: float = 0.1) -> str:
        stp.info(f"Training for {epochs} epochs at lr={learning_rate}")  # Printed by the child, shown here
        return "model.pt"

def crash() -> None:
        raise ValueError("CUDA out of memory")

# Needed because the new process imports this file again
if __name__ == "__main__":
        # Run train(3, learning_rate=0.01) in a fresh process and get its result back
        stp.info("Saved", stp.run_in_subprocess(train, 3, learning_rate=0.01))

        # An error in the child is raised again here, as the same exception
        try:
                stp.run_in_subprocess(crash)
        except ValueError as error:
                stp.warning("The child failed:", error)
Terminal output of the example, the child's log line then its error caught in the parent
wait_for_payload(
result_queue: Queue[JsonDict],
process: Process,
timeout: float | None,
) → JsonDict[source]#

The payload a child puts on its queue, the child being stopped afterwards whatever happened.

The queue is polled every 0.1 seconds, so a KeyboardInterrupt in the parent lands quickly.

Raises:

  • TimeoutError – If no payload came within timeout seconds.

  • RuntimeError – If the child died without sending one.

unpack_payload(
payload: JsonDict,
) → Any[source]#

The value a child returned, or its exception raised again.

The child’s own exception is raised when it survived pickling, chained to a RemoteSubprocessError holding its traceback.

kill_process_tree(process: Process) → None[source]#

Stop a process and every process it started, terminating them first and killing whatever outlives three seconds.

_subprocess_wrapper(
result_queue: Any | None,
func: Callable[[...], R],
args: tuple[Any, ...],
kwargs: dict[str, Any],
capturer: CaptureOutput | None = None,
process_title: str | None = None,
) → None[source]#

Wrapper function to execute the target function and store the result in the queue.

Must be at module level to be pickable on Windows (spawn context).

Parameters:

  • result_queue – Queue to store the result or exception (None if detached).

  • func – The target function to execute.

  • args – Positional arguments for the function.

  • kwargs – Keyword arguments for the function.

  • capturer – Optional CaptureOutput instance for stdout capture.

  • process_title – Optional process title to set.