Futures: Getting Results and Errors Back from a Thread Pool

pool.map is the simplest way to use a thread pool, but it has a weakness. If one call raises an exception, map raises it when you reach that result, and you never see the results after it. To handle each job on its own, use submit and futures.

submit gives you a future

pool.submit(fn, arg) queues fn(arg) to run on a worker thread and immediately gives you back a Future. The call starts as soon as a worker is free. A future is a placeholder for a result that is not ready yet.

from concurrent.futures import Future, ThreadPoolExecutor

with ThreadPoolExecutor(max_workers=4) as pool:
    future: Future[bytes] = pool.submit(download, "https://example.com")
    # ... do other work while the download runs ...
    page = future.result()  # waits until the download finishes

future.result() waits for the job and returns its value. If the job raised an exception, result() raises that same exception in your thread.

Handle results as they finish

concurrent.futures.as_completed(futures) gives you each future as soon as its job finishes. Keep a dictionary from each future to its input, so you know which job each result belongs to:

from concurrent.futures import ThreadPoolExecutor, as_completed

with ThreadPoolExecutor(max_workers=4) as pool:
    jobs = {pool.submit(download, url): url for url in urls}
    for future in as_completed(jobs):
        url = jobs[future]
        try:
            page = future.result()
        except OSError as error:
            print(f"{url} failed: {error}")
        else:
            print(f"{url}: {len(page)} bytes")

One failed download no longer stops the others. You catch each failure at the point where you read its result.

Choose map or submit

• Use pool.map when you want the results in input order, and one failure may stop the whole batch.
• Use submit with as_completed when you want each result as soon as it is ready, or when you need to handle failures one by one.

Always read the result

If you submit a job and never call result(), an exception in that job disappears. Nothing prints, and nothing fails. Read every future you submit, even when the job returns nothing, so that its errors reach you.

Going further

future.exception() returns the job's exception instead of raising it, if you prefer checking to catching. pool.shutdown(cancel_futures=True) drops jobs that have not started yet. The concurrent.futures documentation covers both.