qec/realtime: Relay BP (nv-qldpc) gpu_roce profile for the HSB decoding-server test by cketcham2333 · Pull Request #670 · NVIDIA/cudaqx

cketcham2333 · 2026-07-10T17:16:56Z

Description

Adds a Relay BP (nv-qldpc) profile to the HSB decoding-server test, driving the decoder
through the server's gpu_roce transport — the self-relaunching device-graph scheduler
(GpuRoceTransceiver): enqueue_syndromes / get_corrections / reset_decoder execute as
DEVICE_CALLs on the GPU, and the captured RelayBP decode graph fires device-side
(fire-and-forget + tail self-relaunch) when a window completes. This replaces
hololink_qldpc_graph_decoder_bridge for the decoding-server flow; the old bridge test is
untouched. The pymatching/cpu_roce profile is unchanged and re-verified.

hololink_fpga_syndrome_playback (UNCHANGED —      decoding_server --transport=gpu_roce
 sole FPGA control-plane writer)                    HOLOLINK_* env → GpuRoceTransceiver
  programs SIF w/ server QP/RKey/addr               DEVICE_CALL enqueue/get/reset on GPU,
  writes per-round RPC frames to FPGA BRAM          RelayBP decode graph fired device-side,
  arms ILA, verifies captured RPCResponses          dispatch graph tail-self-relaunches
       FPGA/emulator ──RDMA WRITE──► DOCA GPU rx ring ; server ──RDMA WRITE──► FPGA SIF TX

Changes

hsb_fpga_decoding_server_test.sh

--transport cpu_roce|gpu_roce, defaulted from the decoder profile (pymatching →
cpu_roce, nv-qldpc-decoder → gpu_roce); new --gpu, --proprietary-archive,
--nv-qldpc-plugin flags.
Data generation (fresh each run via surface_code-4-yaml) adds --use-relay-bp for the
nv-qldpc profile and injects transport: gpu_roce into the generated YAML —
DecoderServer selects its transceiver from the per-decoder transport: key, the generator
omits non-default optionals, and the default (cpu_roce) resolves to the not-yet-implemented
CpuRoceTransceiver stub.
gpu_roce server launch: Hololink parameters via HOLOLINK_* env (the server's gpu_roce mode
ignores the cpu_roce CLI flags); remote QP converted to decimal
(GpuRoceConfig::from_env parses base-10 only); CUDA_MODULE_LOADING=EAGER as with the old
bridge launcher; readiness keyed on QEC_DECODING_SERVER_READY gpu_roce (the transceiver
prints the QP/RKey/Buffer handshake before that line, without the bridge banner).
Build phase: builds HSB gpu_roce_transceiver + cuda-quantum cudaq-realtime-bridge-hololink,
wires CUDAQ_QEC_REALTIME_CUDEVICE_PROPRIETARY_ARCHIVE / GPU_ROCE_TRANSCEIVER_LIB /
CUDAQ_REALTIME_BRIDGE_HOLOLINK_LIBRARY into the cudaqx configure, clears the CMake cache
(find_library NOTFOUND staleness), and symlinks the nv-qldpc plugin into
build/lib/decoder-plugins (also opportunistically on --build-less runs). All gated on the
proprietary archive's presence, so pymatching-only rigs build exactly as before.

decoding-server-cqr/CMakeLists.txt — bug fix (separable)
CUDAQ_GPU_ROCE_AVAILABLE was set as CACHE INTERNAL on success, but the plain
set(... FALSE) at the top of the file shadows the cache under CMP0126 (NEW), so the
compile-definition gate for cudaq-qec-decoding-server never fired: the library silently built
without gpu_roce (make_transport threw "CUDAQ_GPU_ROCE_AVAILABLE is not set" at runtime)
while the sibling decoding_server target — which reads the cache — reported
"gpu_roce transport enabled". Fixed by setting the normal variable alongside the cache entry.

…ng-server test Extend hsb_fpga_decoding_server_test.sh to drive the nv-qldpc Relay BP decoder through the decoding server's gpu_roce transport -- the self-relaunching device-graph scheduler (GpuRoceTransceiver): enqueue_syndromes / get_corrections / reset_decoder execute as DEVICE_CALLs on the GPU and the captured RelayBP decode graph fires device-side (fire-and-forget + tail-self-relaunch) when a window completes. This replaces hololink_qldpc_graph_decoder_bridge for the decoding-server flow; the old bridge test is untouched. hsb_fpga_decoding_server_test.sh: - --transport cpu_roce|gpu_roce, defaulted from the decoder profile (pymatching -> cpu_roce HOST_CALL path, nv-qldpc-decoder -> gpu_roce), plus --gpu, --proprietary-archive, --nv-qldpc-plugin. - Data generation (surface_code-4-yaml, fresh each run) adds --use-relay-bp for the nv-qldpc profile and injects `transport: gpu_roce` into the generated YAML: DecoderServer selects its transceiver from the per-decoder transport key, the generator omits non-default optionals, and the default (cpu_roce) resolves to the not-yet-implemented CpuRoceTransceiver stub. - gpu_roce server launch: Hololink parameters go via HOLOLINK_* env (the server's gpu_roce mode ignores the cpu_roce CLI flags), remote QP converted to decimal ($((qp)) -- GpuRoceConfig::from_env parses base-10 only), CUDA_MODULE_LOADING=EAGER as with the old bridge launcher, and readiness keyed on "QEC_DECODING_SERVER_READY gpu_roce" (the transceiver prints the QP/RKey/Buffer handshake before that line, without the bridge banner). - Build phase: builds gpu_roce_transceiver (HSB) and cudaq-realtime-bridge-hololink (cuda-quantum), wires CUDAQ_QEC_REALTIME_CUDEVICE_PROPRIETARY_ARCHIVE / GPU_ROCE_TRANSCEIVER_LIB / CUDAQ_REALTIME_BRIDGE_HOLOLINK_LIBRARY into the cudaqx configure, clears the CMake cache (find_library NOTFOUND staleness), and symlinks the nv-qldpc plugin into build/lib/decoder-plugins. All gated on the proprietary archive's presence so pymatching-only rigs build unchanged. The plugin symlink is also created opportunistically on --build-less runs. decoding-server-cqr/CMakeLists.txt (bug fix, separable): - CUDAQ_GPU_ROCE_AVAILABLE was set as CACHE INTERNAL on success, but the plain `set(... FALSE)` at the top of the file shadows the cache under CMP0126 (NEW), so the compile-definition gate for cudaq-qec-decoding-server never fired: the library was silently built WITHOUT gpu_roce (make_transport threw "CUDAQ_GPU_ROCE_AVAILABLE is not set" at runtime) while the sibling decoding_server target -- which reads the cache -- reported "gpu_roce transport enabled". Set the normal variable TRUE alongside the cache entry. Signed-off-by: Chuck Ketcham <cketcham@nvidia.com>

With CUDAQ_GPU_ROCE_AVAILABLE now actually reaching the library target (the CMP0126 normal-vs-cache shadow fix), cudaq-qec-decoding-server links libdoca_gpunetio.so and libcudaq-realtime-bridge-hololink.so -- both of which reference the CUDA Driver API (cu*). Consumers of the static lib (test_decoding_server_core, test_decoders_yaml, decoding_server) inherit those dependencies, and on driverless build machines (CI containers: CUDA toolkit present, no GPU driver, no libcuda.so.1) their links fail with dozens of `undefined reference to cuStreamCreate` etc. -- the failure seen on all four QEC Build-and-test CI jobs. Link CUDA::cuda_driver alongside CUDA::cudart in the gpu_roce block: the CMake target resolves to the real driver where one is installed and to the toolkit's stubs (lib/stubs/libcuda.so) in driverless environments. This matches the existing precedent in unittests/utils (playback and the old bridge already link CUDA::cuda_driver). Signed-off-by: Chuck Ketcham <cketcham@nvidia.com>

With CUDAQ_GPU_ROCE_AVAILABLE actually reaching the library target (the CMP0126 fix), every consumer of cudaq-qec-decoding-server (test_decoding_server_core, test_decoders_yaml, the CQR plugin) inherited runtime dependencies on libdoca_gpunetio.so and libcudaq-realtime-bridge-hololink.so, which require libcuda.so.1 at LOAD time. On driverless CI machines (CUDA toolkit + HSB + DOCA present, no GPU driver) all four QEC Build-and-test jobs failed: gtest_discover_tests executes the test binaries at build time and the loader cannot satisfy libcuda.so.1 -- link-time driver stubs satisfy ld but not the runtime loader. Restructure so the core library carries no GPU RoCE dependencies while keeping full CI build coverage of the HSB path: - New optional component cudaq-qec-decoding-server-gpuroce holds GpuRoceTransceiver.cpp plus GpuRoceFactory.cpp, the strong definition of the factory (cudaqx_qec_make_gpu_roce_transceiver) that DecodingServer.cpp now references weakly. All DOCA / hololink / CUDA-driver link deps move to this component. - make_transport(gpu_roce) calls the factory when it is linked in and throws a clear "GPU RoCE support is not linked into this binary" error otherwise; the ifdef and the direct GpuRoceTransceiver construction are gone from the core. - The scheduler hookup no longer dynamic_casts to the concrete type: ITransceiver grows a launch_device_scheduler(void*) virtual (default: no device scheduler), overridden by GpuRoceTransceiver to forward to launch_scheduler(). The config-driven constructor keys the hookup on the transport enum instead. - decoding_server links the component WHOLE_ARCHIVE in its proprietary-archive-gated gpu_roce block -- a weak reference does not pull archive members. - A link canary (cudaq-qec-decoding-server-gpuroce-linkcheck; never executed, not installed, not a test) preserves CI link validation of the component: a static archive alone never resolves symbols, and the one real consumer (the daemon's gpu_roce block) is gated on the proprietary cudevice archive, which CI does not provision. The canary forces full link resolution against the hololink / DOCA libraries with the driver stubs -- the same check that used to happen incidentally via the test binaries. - decoding_server's QEC_HAVE_GPU_ROCE_TRANSPORT gate is split from the proprietary-archive gate: the gpu_roce CLI branch references only the (now dependency-free) core DecodingServer API, so it compiles and links whenever HSB/DOCA are present -- CI now builds that branch too. The component + cudevice whole-archive link stays behind the archive gate, so the CI daemon remains loadable on driverless runners. Branch [2a] wraps server startup in try/catch, turning what used to be std::terminate on any gpu_roce bring-up failure into an "ERROR: gpu_roce startup failed: ..." message with exit 1. Net CI coverage: GpuRoceTransceiver.cpp compiles against the HSB/DOCA headers (component builds in `all`), its link resolves (canary), and the daemon's gpu_roce CLI branch compiles and links (split gate). No binary CI executes carries a libcuda.so.1 dependency. On dev rigs the daemon keeps full gpu_roce. As a bonus, the CQR plugin .so is loadable on driverless machines again (it too had inherited the deps). Signed-off-by: Chuck Ketcham <cketcham@nvidia.com>

bmhowe23

Thanks, Chuck. Sharing some comments now ... I'm not sure to what extent we want to handle my comments in this PR vs a follow-up PR. We can discuss offline.

bmhowe23 · 2026-07-10T22:14:23Z

+PROPRIETARY_ARCHIVE="/workspaces/cuda-qx/build/lib/libcudaq-qec-realtime-cudevice-proprietary.a"
+NV_QLDPC_PLUGIN="/workspaces/cuda-qx/build/lib/decoder-plugins/libcudaq-qec-nv-qldpc-decoder.so"


The .a file and .so files should be installed/delivered artifacts. These bash env vars should not be pointing to a private build area. If the plan is to put these in the Docker container as example tests, the paths should be to the installed area. @anjbur for viz.

bmhowe23 · 2026-07-10T22:16:30Z

+    # hololink_core is enough for playback + the emulator; the gpu_roce
+    # server additionally needs the HSB gpu_roce_transceiver archive.
+    local hsb_targets=(hololink_core)
+    local hsb_gpu_roce_lib="${hsb_build}/src/hololink/operators/gpu_roce_transceiver/libgpu_roce_transceiver.a"


@anjbur for viz. @cketcham2333 does this mean that the decoding server docker image needs to have the HSB source code? Is the .a in that source code? I'm just trying to think about how this is productized. (This comment assumes that we want this script to be included as some sort of a self-test include in the container.)

bmhowe23 · 2026-07-10T22:17:28Z

+
+    # Clear stale cmake cache entries (find_library caches NOTFOUND
+    # permanently, and CUDAQ_GPU_ROCE_AVAILABLE is CACHE INTERNAL).
+    rm -f "$cudaqx_build/CMakeCache.txt"


Is this still needed? This seems a bit kludgy.

bmhowe23 · 2026-07-10T22:19:01Z

+    # Clear stale cmake cache entries (find_library caches NOTFOUND
+    # permanently, and CUDAQ_GPU_ROCE_AVAILABLE is CACHE INTERNAL).
+    rm -f "$cudaqx_build/CMakeCache.txt"
+


The cmake and build of cudaqx itself is making me reconsider whether this script is the right thing to put in the Docker container. That is - ideally we wouldn't have to have a cudaqx build script running in an already prebuilt Docker container for the decoding server.

cketcham2333 added 2 commits July 10, 2026 17:10

Merge remote-tracking branch 'upstream/main' into add-relaybp-fpga

e7ecb38

cketcham2333 marked this pull request as ready for review July 10, 2026 17:19

cketcham2333 added 2 commits July 10, 2026 17:50

cketcham2333 requested review from bmhowe23, melody-ren and vedika-saravanan July 10, 2026 21:42

bmhowe23 reviewed Jul 10, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

qec/realtime: Relay BP (nv-qldpc) gpu_roce profile for the HSB decoding-server test#670

qec/realtime: Relay BP (nv-qldpc) gpu_roce profile for the HSB decoding-server test#670
cketcham2333 wants to merge 4 commits into
NVIDIA:mainfrom
cketcham2333:add-relaybp-fpga

cketcham2333 commented Jul 10, 2026

Uh oh!

bmhowe23 left a comment

Uh oh!

bmhowe23 Jul 10, 2026

Uh oh!

bmhowe23 Jul 10, 2026

Uh oh!

bmhowe23 Jul 10, 2026

Uh oh!

bmhowe23 Jul 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		PROPRIETARY_ARCHIVE="/workspaces/cuda-qx/build/lib/libcudaq-qec-realtime-cudevice-proprietary.a"
		NV_QLDPC_PLUGIN="/workspaces/cuda-qx/build/lib/decoder-plugins/libcudaq-qec-nv-qldpc-decoder.so"

Uh oh!

Conversation

cketcham2333 commented Jul 10, 2026

Description

Changes

Uh oh!

bmhowe23 left a comment

Choose a reason for hiding this comment

Uh oh!

bmhowe23 Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

bmhowe23 Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

bmhowe23 Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

bmhowe23 Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants