[BUG] distributed ring: SocketThread dies silently on transient connection reset — all ranks wedge forever in Event::wait #3862

Summary

In mlx/distributed/ring/ring.cpp, a transient TCP error (e.g. ECONNRESET from a WiFi hiccup or NAT idle-timeout) makes the affected SocketThread::worker() return after 10 failed attempts — without resolving its pending std::promises and without any error propagation. Every rank of the ring then blocks forever in mx.eval → array::wait → Event::wait. At the Python layer nothing ever throws; a distributed job just freezes permanently.

Observed on a heterogeneous 5-node ring (2× macOS/Metal, 3× Linux/CUDA — exo cluster, cross-linked below). Smoking gun from the affected rank's stderr:

[ring] Receiving from socket 51 failed with errno 54    (×10 within ~1 ms)
[ring] Too many send/recv errors. Aborting...

errno 54 (macOS) = ECONNRESET. After this, the whole ring is wedged; TCP byte counters on all ring sockets freeze. The probability of hitting this scales with world size and link quality, which initially disguised it as an "N=5 fails, N≤4 works" bug.

Code path (v0.32.0)

1. SocketThread::worker() — on error_count >= 10:

   if (error_count >= 10) {
     log_info(true, "Too many send/recv errors. Aborting...");
     return;   // pending promises in sends_/recvs_ are never resolved
   }

2. All internal future consumers use .wait() (all_reduce_impl, all_gather_impl, send, recv) — even if the promises were failed, nothing would throw.
3. Latent secondary bug: in the recv path, r == 0 (orderly peer close) falls into else if (errno != EAGAIN) with a stale errno. If errno happens to be EAGAIN, a closed socket spins forever and doesn't even increment error_count.

Suggested fix (tested locally, happy to open a PR)

• On worker abort: set a dead_ flag under the queue mutex and resolve all pending promises with set_exception; make send_impl/recv_impl fail immediately when dead_.
• Switch the internal .wait() calls to .get() so the exception propagates and the job fails loudly (an orchestrator can then recreate the group — self-healing instead of a permanent silent hang).
• Count r == 0 as an error in the recv path.
• Set SO_KEEPALIVE on ring sockets (reduces NAT/idle resets and detects dead peers).
• (Longer term: transparent reconnect would be even better, but fail-fast already turns an unrecoverable hang into a recoverable error.)

Environment

mlx 0.32.0 (official macOS wheels + CUDA source builds), ring backend over TCP, 5 nodes (M5/M4 Max, GB10, Jetson Thor sm_110, RTX 5090 sm_120/WSL2). Cross-link with full investigation history: exo-explore/exo#2219