Split Work into Chunks for a Process Pool

A process pool sends work to its workers in messages. If you send 10,000 small jobs one at a time, the pool sends 10,000 messages, and the cost of sending them can be bigger than the work itself. The fix is to send the jobs in chunks, which are groups of jobs that the pool sends to a worker in one message. Write the function that splits a list into chunks.

chunks([1, 2, 3, 4, 5, 6, 7], 3)
# [[1, 2, 3], [4, 5, 6], [7]]

What to write

Fill in chunks(items, size):

• Return a list of lists. Each inner list holds the next size items, in their original order.
• The last chunk holds whatever is left, so it can be shorter than size.
• An empty items gives an empty list.
• If size is less than 1, raise ValueError.

What the driver checks

Keep solve as it is. It returns:

• chunks: the list your function returned.
• messages: the number of chunks, which is how many messages a pool would send for this work.
• rejected: True if your function raised ValueError. Then chunks is empty and messages is 0.

ProcessPoolExecutor.map takes a chunksize argument that does this for you. The next article explains how to choose it.