We can't find the internet
Attempting to reconnect
Something went wrong!
Attempting to reconnect
← Parallel Python with Process Pools step 5 of 24
Give Each Worker Its Own Database Connection
Your app opens a database connection at startup and keeps it in a global.
Under fork, each worker gets a copy, so the parent and the worker share
one socket. Under spawn or forkserver, the global is None in
every worker. The fix is to send each worker a recipe, a small picklable
description such as the connection string. An initializer then builds the
worker's own connection from it:
recipe = ("pg://db",)
with ProcessPoolExecutor(initializer=worker_init, initargs=(recipe,)) as pool:
results = list(pool.map(worker_handle, rows))
The grader cannot start processes, so solve simulates the pool. It runs
each worker's lifetime one after another. The starter's Conn is a fake
connection. It holds a lock, so pickle rejects it, and it counts the rows
it serves.
What to write
Recipe is a one-element tuple holding the connection string, (dsn,).
worker_init(recipe)builds a newConnfrom the recipe, stores it in_CONN, and adds one to_OPENED.worker_handle(row)returns the current connection'squery(row). If there is no connection, it raisesRuntimeError("worker_handle called before worker_init").
solve(*, dsn, rows, workers) runs these steps:
- Set
_CONNtoNoneand_OPENEDto0. - If
workers <= 0, raiseValueError("workers must be positive"). - Give worker
wthe rows at indexesw,w + workers,w + 2 * workers, and so on. Run the workers in order from 0. - Skip a worker with no rows. For each other worker, call
worker_initonce, handle its rows in order, then set_CONNback toNone. Reset_CONNeven when a row raises, and let the error reach the caller. - Return
(results, opened).resultsis a tuple in the original row order.openedis how many connectionsworker_initbuilt.
Track each row's index, so two equal rows stay separate. Don't change
rows. Empty rows with a positive workers returns ((), 0).
Example
Conn.query returns f"{dsn}#{serial}:{row.upper()}", where serial is 1
for a connection's first row, 2 for its second, and so on. With dsn="db", rows a, b, c, d and two
workers:
(("db#1:A", "db#1:B", "db#2:C", "db#2:D"), 2)
Worker 0 serves a and c. Worker 1 serves b and d. Each worker reuses
its own connection, so c is the second row on worker 0's connection.
Pass mypy --strict
Annotate the global as _CONN: Conn | None. In worker_handle, copy
_CONN into a local variable, raise if it is None, then call query on
the local. mypy then knows the local is a Conn.
Going further
A real pool decides which worker gets which row, so a real run won't match
this split or these serial numbers. Close a real connection with its own
close() method. Setting the global to None is enough only for this fake.
Your code, saved here
Sign in to access your accepted solutions and saved draft history. You can keep reading the lesson without an account.
Sign in to your accountStuck?
Python reference solution
Sign in to track your attempts toward unlocking the reference solution.