ThreadPoolExecutor Context Manager Produces Unexpected Results When Using Future.add_done_callback

[PyWoody]

Bug report

Bug description:

Using a context manager with the concurrent.futures.ThreadPoolExecutor produces different results depending on how the context is exited.

As shown below, if you wait until the context is exited naturally, all Future.add_done_callback's are processed; if you return within the context, not all callbacks are processed; and, if you manually call shutdown(wait=True) before returning in the context, the callbacks are processed as expected. Please see examples of each situation reproduced below. As a side note, I ran the same functions with 300 instead of 30 iterations to check if it was a race condition issue but they returned 300, 1, 300, which is in line with 30, 1, 30.

The second example, namely, shutdown_default_with_return, is counter-intuitive to how I expected the context manager to function. As shown on Line 666 of concurrent.futures._base.Executor.__base__, self.shutdown(wait=True) is called before exiting the context. The documentation on concurrent.futures.Exector.shutdown also states:

If wait is True then this method will not return until all the pending futures are done executing and the resources associated with the executor have been freed.

If the base class is calling shutdown with self.shutdown(wait=True), why are all callbacks not processed if manually calling the same method with the same arguments processes all callbacks?

 

# setup
import time

from threading import Lock
from concurrent.futures import ThreadPoolExecutor

LOCK = Lock()

# returns 30
def shutdown_default():
    result = 0

    def increase(future):
        nonlocal result
        with LOCK:
            result += 1

    with ThreadPoolExecutor(max_workers=3) as executor:
        for i in range(30):
            f = executor.submit(time.sleep, i / 100)
            f.add_done_callback(increase)
    return result

# returns 0 or 1
def shutdown_default_with_return():
    result = 0

    def increase(future):
        nonlocal result
        with LOCK:
            result += 1

    with ThreadPoolExecutor(max_workers=3) as executor:
        for i in range(30):
            f = executor.submit(time.sleep, i / 100)
            f.add_done_callback(increase)
        return result

# returns 30
def shutdown_manual():
    result = 0

    def increase(future):
        nonlocal result
        with LOCK:
            result += 1

    with ThreadPoolExecutor(max_workers=3) as executor:
        for i in range(30):
            f = executor.submit(time.sleep, i / 100)
            f.add_done_callback(increase)
        executor.shutdown(wait=True)
        return result

>>> shutdown_default()
30
>>> shutdown_default_with_return()
1
>>> shutdown_manual()
30

CPython versions tested on:

3.14

Operating systems tested on:

macOS

[PyWoody]

Closing this as the expected functionality happens when working with mutable objects.

[picnixz]

• 1st example: the context manager waits until everything is done and then exits, so return result is really done after everything is completed.
• 3rd example: same, you ask for the context manager to terminates and then return. So you properly wait for the tasks to complete.

The 2nd example is more tricky but you can see that the result is incremented:

def shutdown_default_with_return():
    result = 0

    def increase(future):
        nonlocal result
        print(result)  # you'll see it increments
        with LOCK:
            result += 1

    with ThreadPoolExecutor(max_workers=3) as executor:
        for i in range(30):
            f = executor.submit(time.sleep, i / 100)
            f.add_done_callback(increase)
        return result

I think the reason why sometimes you have 0 and sometimes 1 depends on what the value is when hitting return result. When hitting the return, we store the ref to the immutable result and return its value. With a list, as you said, you'll the difference but I think it's because we're actually returning a ref to it.

@Fidget-Spinner Can you confirm whether this is the expected behavior?

[PyWoody]

@picnixz , Yes, that's my interpretation of the behavior as well. I think the only bug here was between the ears.

[Fidget-Spinner]

@Fidget-Spinner Can you confirm whether this is the expected behavior?

I don't know what is the expected behavior of TheradPoolExecutor.

[picnixz]

Sorry, I should have been clearer, I just wanted to be sure that the interpreter, in the following examples, behaves as I expected, that is:

def f():
	result = 0
	try:
	    return result  # will be 0
	finally:
	    result += 1

vs

def f():
	x = []
	try:
	    return x  # will be [1]
	finally:
	    x.append(1)

However, now that I've unrolled everything, it seems clear to me that it was the expected behavior (albeit sometimes a bit confusing at first glance). So, I think what happens with context managers in general is the above (in our case, the callbacks are scheduled and called but they don't affect the returned value as they are in the "finally" part).

[Fidget-Spinner]

Oh wow, that is indeed very surprising behavior!