stouputils.parallel module

This module provides utility functions for parallel processing, such as:

multiprocessing(): Execute a function in parallel using multiprocessing
multithreading(): Execute a function in parallel using multithreading

I highly encourage you to read the function docstrings to understand when to use each method.

multiprocessing(func: Callable[[T], R], args: list[T], use_starmap: bool = False, chunksize: int = 1, desc: str = '', max_workers: int = 4, delay_first_calls: float = 0, verbose: int = 0) → list[R][source]

Method to execute a function in parallel using multiprocessing, you should use it:

For CPU-bound operations where the GIL (Global Interpreter Lock) is a bottleneck.
When the task can be divided into smaller, independent sub-tasks that can be executed concurrently.
For operations that involve heavy computations, such as scientific simulations, data processing, or machine learning tasks.

Parameters:

func (Callable) – Function to execute
args (list) – List of arguments to pass to the function
use_starmap (bool) – Whether to use starmap or not (Defaults to False): True means the function will be called like func(*args[i]) instead of func(args[i])
chunksize (int) – Number of arguments to process at a time (Defaults to 1 for proper progress bar display)
desc (str) – Description of the function execution displayed in the progress bar
max_workers (int) – Number of workers to use (Defaults to CPU_COUNT)
delay_first_calls (float) – Apply i*delay_first_calls seconds delay to the first “max_workers” calls. For instance, the first process will be delayed by 0 seconds, the second by 1 second, etc. (Defaults to 0): This can be useful to avoid functions being called in the same second.
verbose (int) – Level of verbosity, decrease by 1 for each depth

Returns:

Results of the function execution

Return type:

list[object]

Examples

>>> multiprocessing(doctest_square, args=[1, 2, 3])
[1, 4, 9]

>>> multiprocessing(int.__mul__, [(1,2), (3,4), (5,6)], use_starmap=True)
[2, 12, 30]

>>> # Will process in parallel with progress bar
>>> multiprocessing(doctest_slow, list(range(10)), desc="Processing", verbose=1)
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

>>> # Will process in parallel with progress bar and delay the first threads
>>> multiprocessing(doctest_slow, list(range(10)), desc="Processing with delay", max_workers=2, delay_first_calls=1.2, verbose=1)
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

multithreading(func: Callable[[T], R], args: list[T], use_starmap: bool = False, desc: str = '', max_workers: int = 4, delay_first_calls: float = 0, verbose: int = 0) → list[R][source]

Method to execute a function in parallel using multithreading, you should use it:

For I/O-bound operations where the GIL is not a bottleneck, such as network requests or disk operations.
When the task involves waiting for external resources, such as network responses or user input.
For operations that involve a lot of waiting, such as GUI event handling or handling user input.