Skip to content

← Parallel Python with Process Pools step 20 of 24

Hard End-to-End

Simulate a Pool Shutdown with a Grace Period

Your batch service runs jobs on a pool of worker processes. During a deploy, the platform sends it a SIGTERM signal and kills it a few seconds later. A graceful shutdown starts no new jobs, gives running jobs a grace period to finish, and stops the rest when that period ends. The site's grader cannot start worker processes, so you simulate this plan with a clock that counts whole ticks.

solve(names=["a", "b", "c", "d"], durations=[3, 5, 1, 4],
      workers=2, sigterm_at=4, grace=2)
# (("a", "c", "b"), ("d",), ("sigterm-received", "stop-accepting", "drain-complete"))

Two workers start a and b at tick 0. a finishes at tick 3, so c starts and finishes at tick 4. SIGTERM arrives at tick 4, so d never starts. b finishes at tick 5, inside the grace period. With grace=0, the deadline is tick 4. c still counts as completed, the deadline stops b, and the second tuple is ("b", "d").

The rules

  • names and durations have the same length. Two jobs can share a name, so tell jobs apart by their position in the input.
  • Durations, sigterm_at and grace are whole numbers, 0 or more. workers is at least 1.
  • The clock starts at tick 0. A job that starts at tick t with duration d finishes at tick t + d. At most workers jobs run at once.
  • The deadline is tick sigterm_at + grace.

At each tick where something happens, take these steps in order:

  1. Record the jobs that finish now, in the order they started.
  2. From tick sigterm_at on, start no more jobs.
  3. At the deadline, stop every job still running. A job that finishes exactly at the deadline is already recorded, so it counts as completed.
  4. Before tick sigterm_at, start jobs in input order on the free workers. A job with duration 0 finishes the moment it starts, which frees its worker for the next job at the same tick.

The simulation ends when every job has finished, when the running jobs have all finished after SIGTERM, or at the deadline. Jump the clock to the next event instead of stepping through every tick.

What to return

Return a tuple of three tuples:

  1. The names of completed jobs, in the order they finished.
  2. The names of jobs that never started or that the deadline stopped, in input order.
  3. The actions, exactly one of these:
    • ("completed-before-signal",) if every job finished before tick sigterm_at.
    • ("sigterm-received", "stop-accepting", "drain-complete") if SIGTERM arrived and every running job finished by the deadline.
    • ("sigterm-received", "stop-accepting", "grace-expired", "terminate-workers") if the deadline stopped any job.

If the last job finishes exactly at tick sigterm_at, the signal arrives at the same tick, so the actions start with "sigterm-received". With sigterm_at=0, no job starts, and the actions end with "drain-complete". An empty job list finishes at tick 0, so it gets ("completed-before-signal",) unless sigterm_at is 0.

Going further

When you stop a real job, whatever it already wrote stays written, so retry it only if running it twice is safe. With a ProcessPoolExecutor, shutdown(wait=False, cancel_futures=True) rejects new jobs and cancels queued ones. The executor has no deadline, so enforce it yourself. Wait with concurrent.futures.wait(running, timeout=grace), then call terminate_workers(), which Python 3.14 added.