SDK: pipelineStore() for bulk uploads#420
Open
x3c41a wants to merge 3 commits into
Open
Conversation
x3c41a
force-pushed
the
ndk/sdk-pipeline-submitter
branch
2 times, most recently
from
85da58e to
1de6a4f
Compare
x3c41a
changed the title
x3c41a
force-pushed
the
ndk/sdk-pipeline-submitter
branch
2 times, most recently
from
87d23b2 to
d9b4f4a
Compare
x3c41a
added a commit
that referenced
this pull request
Apr 27, 2026
Two-panel layout matching #420: left = finalized throughput (tx/s + KB/s), right = stacked latency where each colour band shows the additional time at the next percentile (p50 / p90-p50 / p95-p90 / p99-p95). Defaults to finalization latency; --latency inclusion switches to broadcast-to-best-block.
Adds pipelineStore() to the TypeScript SDK. Drives bulk submission from a single chainHead subscription on the first RPC and broadcasts each signed tx to every configured RPC, gating completion on finalization rather than pool acceptance. Optimisations: * Offline API: metadata is decoded once at init via getOfflineApi rather than on every signTx call. * Fast signer: signed extensions are pre-decoded once; per-tx signing bypasses PAPI's ~100ms decAnyMetadata work and stays under 5ms. * Speculative pre-signing: while one batch broadcasts, the next batch is signed concurrently so the predicted-nonce path is broadcast-only on the next bestBlockChanged. * Mortality period 64 blocks: short eras (8) deadlock the pool when a tx expires before inclusion but blocks new submissions at the same nonce. Per-item latency tracking: each broadcast records its first-broadcast timestamp; bestBlockChanged and finalized handlers attribute the inclusion / finalization observation back to those timestamps. The result exposes count / min / max / mean / p50 / p90 / p99 plus the raw arrays on PipelineResult so callers can build their own distributions. LatencyStats and the rest of the pipeline types are re-exported from sdk/typescript/src/index.ts.
x3c41a
force-pushed
the
ndk/sdk-pipeline-submitter
branch
2 times, most recently
from
270267c to
8bba8a6
Compare
pipeline-stress.ts is a standalone benchmark harness. It authorises a fresh derived submitter, generates the requested payloads, runs pipelineStore, prints the latency summary (p50 / p90 / p99 plus min / max / mean), and writes the full PipelineResult plus raw latency arrays to a JSON file via --output-json. plot-pipeline-results.py reads one or more of those JSON files and produces a 2-panel chart: finalised throughput on the left (tx/s and KB/s) and stacked finalisation latency on the right (each colour band is the additional latency from p50 -> p90 -> p99). --latency inclusion switches the right panel to broadcast -> best-block. The committed pipeline-results.png is a payload sweep against Versi (4 RPCs, single account): | items | size | tx/s | KB/s | final p50 | final p99 | |------:|-------|------:|-----:|----------:|----------:| | 2000 | 1 KB | 30.44 | 30 | 31.0s | 44.7s | | 1000 | 10 KB | 25.69 | 257 | 31.5s | 33.8s | | 500 | 100 KB| 6.79 | 679 | 39.7s | 48.2s | | 500 | 128 KB| 4.92 | 630 | 40.9s | 70.9s | | 250 | 256 KB| 2.60 | 665 | 33.6s | 70.3s | | 150 | 512 KB| 1.48 | 756 | 43.9s | 74.1s | | 100 | 1 MB | 0.67 | 683 | 69.2s | 89.7s | Bandwidth peaks around 512 KB (~756 KB/s) which is where length and count budgets balance. Below 100 KB the count cap (512 tx/block) idles the length budget; above 256 KB length pressure starts trimming the batch. Result JSON files live in sdk/typescript/test/stress/results/ and are gitignored.
x3c41a
force-pushed
the
ndk/sdk-pipeline-submitter
branch
from
8bba8a6 to
fd36a49
Compare
x3c41a
changed the title
rosarp
reviewed
May 12, 2026
Member
There was a problem hiding this comment.
if this is generated file, this should be git ignored
Collaborator
|
I will take over this branch and continue the topic - my main idea is to make the pipelineStore solution basically a default implementation of "store" for the SDK |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
pipelineStore()is Bulletin SDK's bulk-upload function.It pushes a single account to its ~700 KB/s ceiling on Bulletin Chain.
It handles nonces, pool, mortality, finalisation and reports per-item latency.
Benchmark by payload sizes (Versi, 4 RPCs, single account)
The sweet spot is 512 KB at ~756 KB/s. Smaller payloads hit the 512 tx-per-block limit before they fill a block. Larger payloads are big enough that fewer fit per block. Both leave bandwidth on the table.
Algorithm and key optimisations
On each best block: query nonce, compute the largest batch that fits one block (weight + length + count), sign each tx with a 64-block era, broadcast to all RPCs. Completion gates on finalisation.
getOfflineApi): metadata is decoded once at init, not per-tx.decAnyMetadatawork and stays under 5ms.PipelineResultas count / min / max / mean / p50 / p90 / p99 plus the raw arrays.Repro