set_bad_state_and_fail loop ends early if race condition on future fires

[benclifford]

Describe the bug

BlockProviderExecutor.set_bad_state_and_fail loops over outstanding tasks setting an exception. If that exception set fails, because the future has its state set already, then the loop ends immediately due to an exception raised from future.set_exception. Tasks later in the sequence will not be set as failed.

Nothing ensures that set_bad_state_and_fail is consistent with anything else happening on futures - for example, in #3992 the job status poller continues to call set_bad_state_and_fail repeatedly, with subsequent runs failing.

or hypothetically: if a task completes successfully at "the same time" as set_bad_state_and_fail (for example, if a batch system submit command was working earlier and now is not, with the windowed_error_handler), then I think this behaviour can fire, resulting in subsequent tasks not being marked as failed.

Errors are logged about this happening, in parsl.log.

To Reproduce
Reproduce #3992

Expected behavior

Environment
Python 3.12, parsl 0d4a261

Distributed Environment
my laptop