Follow up fixes for pinning hardware device#678
Draft
melody-ren wants to merge 5 commits into
Draft
Conversation
Decoders can now be pinned to a specific CUDA device at construction via a cuda_device_id parameter, settable through C++/Python kwargs and the YAML realtime config (top-level decoder field, next to type/ transport). Model: one thread owns one decoder. decoder::get() validates the id (negative or >= device count throws), persistently pins the constructing thread with cudaSetDevice (no restore), strips the key before the plugin constructor, and stores it (get_cuda_device_id()). Plugin constructors therefore allocate on the right device with zero plugin changes -- this covers trt_decoder and the closed-source nv-qldpc-decoder transparently. decode_async() is the one exception: its fresh std::async worker pins itself for the call's duration via a lib-private RAII CudaDeviceGuard (libs/qec/lib/hardware_guards.h). The realtime host dispatcher (one thread serving all decoders) applies each decoder's device with set-if-different before enqueue and before DEVICE-mode graph capture; no-op for unpinned decoders. Tests: 7 C++ unit tests incl. 2-GPU placement and async-worker pinning, YAML round-trip + prepare_decoder_params coverage, Python kwargs tests. NUMA/mempolicy/cpu_affinity and thread-binding APIs are deferred to a follow-up PR per review feedback on NVIDIA#634. Signed-off-by: kvmto <kmato@nvidia.com>
cuda_device_id pinned only the constructing thread, so two dispatch paths decoded on whatever device was current: - decoding-server DecodingSession workers start unpinned; every session decoded on the default device regardless of its pin. The worker now pins itself once at worker_loop entry. - the direct (no realtime session) path decodes on the caller thread, which configure_decoders leaves on the LAST decoder's device. It now selects the decoder's device before each decode. Both paths share a new fail-fast helper (hardware_guards.h, set-if-different, throws). apply_decoder_cuda_device also uses it now: a cudaSetDevice failure previously warned and continued on the wrong device; it now surfaces as a dispatch error response, and graph capture aborts during initialization. Signed-off-by: Melody Ren <melodyr@nvidia.com>
Two paths could previously run with a decoder's cuda_device_id silently violated -- the worst failure mode, because decoding can still succeed on the wrong GPU with nothing but an unread log line as evidence: - DecodingSession workers logged a cudaSetDevice failure and kept serving. The pin now runs on the worker thread behind a promise/future handshake in start_worker(): the worker either starts pinned or the exception is rethrown on the registry thread and server startup aborts, the same channel as a decoder that fails to construct. - gpu_roce chose its GPU from HOLOLINK_GPU_ID (default 0) with no knowledge of the decoder's pin, splitting graph capture (pinned device) from ring buffers and device-side graph launch (env device). Both knobs name the same topology fact -- the FPGA-affine GPU -- so they are now reconciled at transport creation: agreement or a single set knob resolves the device, a conflict throws, and an unset environment defers to the decoder's pin. The reconciliation is a plain function compiled in every configuration and unit-tested; the gpu_roce call site remains behind CUDAQ_GPU_ROCE_AVAILABLE and needs hardware validation. Signed-off-by: Melody Ren <melodyr@nvidia.com>
- set_cuda_device_for_decode: -1 is a no-op and an impossible device id throws -- both runnable on GPU-less CI (cudaSetDevice past the device count fails there too), covering the failure transport the worker handshake rides on, which cannot be induced end-to-end after construction-time validation. - DecodingSession handshake smoke: a decoder pinned to device 0 starts its worker through the promise/future handshake and serves a queued item (skips below 1 GPU). Signed-off-by: Melody Ren <melodyr@nvidia.com>
melody-ren
force-pushed
the
melodyr/pin-device-fixes
branch
from
e0fda65 to
310fdff
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary:
This is a follow-up PR to #664, which should be merged before this PR