fix: use explicit spawn context for all multiprocessing primitives

[dodamih]

Queue, Lock, and ProcessPoolExecutor were using different mp contexts. mp.Queue() and mp.Lock() used the default context (fork on Linux), while ProcessPoolExecutor used an explicit spawn context. Passing fork-context primitives to spawn-context workers raises RuntimeError.

This worked inconsistently because cloudfiles calls multiprocessing.set_start_method("spawn", force=True) as a side effect during its first download. If that ran before parallel_execution, the default context was already spawn and it matched. Otherwise it crashed.

Fix: create all primitives from the same explicit spawn context.

Note the following subtlety:

If a CV has a mesh directory, during CloudVolume.__getitem__, it calls PrecomputedMeshMetadata.fetch_info() → cache.download_json() → cloudfiles.get() → cloudfiles.parallel_execute → set_start_method("spawn", force=True). This happens before parallel_execution runs - so it works fine; it only manifests for volumes without meshes.

Also, CloudFiles really should not force change the global start_method for a download.

[william-silversmith]

Talking with Nico about this in https://github.com/seung-lab/python-task-queue/pull/47/changes

[dodamih]

Thanks - hopefully whatever you end up doing there can also be applied here.

[dodamih]

Also, you already have a similar fix in #659 in wms_refactor_shared_memory branch.

[william-silversmith]

Good point! Why not ctx.cpu_count() too then? I feel more confident about this then since I tested it myself. 😁 The issue with that other PR wasn't the multiprocessing (after I fixed it) it was some warning message I couldn't get rid of related to shared memory.