Skip to content

← Error Handling with Exceptions, Retries and Warnings step 11 of 20

Medium End-to-End

Collect Every Failure in a Batch

A nightly job processes a batch of records, one after another. When a record fails, the job keeps going, collects the error, and reports every error at the end. If every record fails, it raises the errors together in an ExceptionGroup.

records: [{"id": "r1", "action": "ok"}, {"id": "r2", "action": "transient"}]
result:  {"outcome": "report", "succeeded": ("r1",), "failed": ("r2",),
          "errors": [{"type": "TransientError", "message": "r2 timed out",
                      "notes": ["record=r2"]}],
          "attempted": ["r1", "r2"]}

Write process_batch

process_batch(records, handler) calls handler(record) once for each record, in order. Every record has a string "id".

  • If the handler returns, add the id to succeeded.
  • If it raises an Exception, call add_note(f"record={record_id}") on that exception, keeping any notes it already has. Then add the exception to errors and the id to failed.
  • If it raises any other BaseException, such as KeyboardInterrupt, let it escape at once, unchanged, and stop the batch.
  • Keep each list in the order things happened, duplicates included.

At the end, if there are errors and succeeded is empty, raise ExceptionGroup("every record in the batch failed", errors). Otherwise return a BatchReport whose three fields are tuples. An empty batch returns three empty tuples.

BatchReport is a dataclass. @dataclass writes its __init__ for you, so BatchReport(succeeded, failed, errors) builds one. frozen=True stops later code from changing its fields.

Write solve

solve(records) runs process_batch with a handler that first appends the record's id to an attempted list and then reads its "action":

Action What the handler does
"transient" raises TransientError(f"{record_id} timed out")
"terminal" raises TerminalError(f"{record_id} is malformed")
"abort" raises KeyboardInterrupt(f"{record_id} interrupted the worker")
anything else, or no action returns

One try takes either except or except* clauses, so nest two try statements. The inner try has except* TransientError and except* TerminalError, and both can run for one group. The outer try catches BaseException to report an abort. An except* clause cannot return, so store results in variables and return after the try statement.

Each except* clause receives a group of matching errors. Describe them in order. A member can itself be a group, so walk into it recursively, depth first, and describe each single error you reach.

What solve returns

{"outcome": "report", "succeeded": (...), "failed": (...), "errors": [...], "attempted": [...]}
{"outcome": "group", "transient": [...], "terminal": [...], "attempted": [...]}
{"outcome": "aborted", "abort_type": <class name>, "attempted": [...]}

Use "report" when process_batch returns, even for an empty batch. Use "group" when you handle the all-failed group, with [] for a kind that has no errors. Use "aborted" when an exception escapes both except* clauses, with its class name as "abort_type" and no earlier failures.

Describe each error as {"type": ..., "message": ..., "notes": [...]}, using its class name, str(exc), and its list of notes, which may be empty. Return succeeded and failed as plain tuples. The test runner shows tuples in a tagged form, and it converts them for you.