Graceful Shutdown of a Process Pool

A pool that ignores SIGTERM gets SIGKILLed mid-write. What you find afterwards is half-written output files, a corrupted queue, and leaked POSIX semaphores that nothing will clean up. Kubernetes gives you terminationGracePeriodSeconds; what you do with it is a design decision most codebases never make.

The escalation ladder, with its documented warnings

Process.terminate() — SIGTERM. The docs are blunt about the cost:

exit handlers and finally clauses will not be executed

and descendants of the process are orphaned, and:

if the process has acquired a lock or semaphore… terminating it is liable to cause other processes to deadlock

which leads to the strongest sentence in the module’s documentation: “only consider using terminate() on processes which never use any shared resources.”

Process.kill() — SIGKILL. Everything above, minus any chance of cleanup.

Process.interrupt() — new in 3.14, and the rung that was missing. It raises KeyboardInterrupt in the child, so finally blocks and context managers actually run. The caveat is honest: “if the child process catches and discards KeyboardInterrupt, the process will not be terminated” — cooperative means cooperative.

Executor.shutdown(wait=True, cancel_futures=True) — stop accepting, cancel what has not started, let the in-flight work finish.

ProcessPoolExecutor.terminate_workers() / kill_workers() — 3.14, the pool-level equivalents of the first two rungs.

The correct sequence is: stop accepting -> drain with a deadline -> escalate. Never jump straight to the bottom of the ladder.

Your task

The behaviour under test is an ordering, not a duration, so model it as one. Ticks are the unit; a task with duration d started at tick t completes at tick t + d.

def solve(
    *,
    names: list[str],
    durations: list[int],
    workers: int,
    sigterm_at: int,
    grace: int,
) -> tuple[tuple[str, ...], tuple[str, ...], tuple[str, ...]]:

Return (completed in completion order, work to re-submit, actions taken).

Rules:

• At most workers tasks run at once. Idle slots take the next un-started task, in input order.
• Within a tick: completions are recorded first, then new work starts.
• Ties in completion time break by start order.
• From sigterm_at onward, no new task starts. In-flight work continues.
• At sigterm_at + grace, anything still running is terminated.
• Work to re-submit = never started plus terminated in flight, in input order. A task killed at the deadline produced nothing, so it is work you still owe.

Actions, exactly:

• Finished before the signal ever fired: ("completed-before-signal",)
• Drained inside the grace window: ("sigterm-received", "stop-accepting", "drain-complete")
• Ran out of grace: ("sigterm-received", "stop-accepting", "grace-expired", "terminate-workers")

The property that matters most: results already produced are never lost, and the function returns rather than hanging. A shutdown path that can hang is worse than no shutdown path, because the orchestrator’s own timeout will SIGKILL you and you are back where you started.

The typing lessons

Two small things that a --strict build will not let you fudge.

A signal handler has one legal shape:

def _on_term(signum: int, frame: FrameType | None) -> None: ...
signal.signal(signal.SIGTERM, _on_term)

FrameType comes from types, and the | None is real — the frame is None when the signal arrives in C code.

And the shutdown flag shared with workers is multiprocessing.synchronize.Event, never multiprocessing.Event — which is a bound method and not a type at all. That is the lesson from “Typing Multiprocessing” arriving where you actually need it.