Skip to content

Commit 03f44fe

Browse files
docs: add BBR congestion control tuning for Jellyfin streaming (#67)
TCP cubic caused multi-second stalls on the 200ms Tailscale path due to Jio router buffer overflow. BBR eliminates stalls and improves throughput from 8 Mbps to 25 Mbps stable. Documents the fix, verification steps, and rollback procedure. Co-authored-by: Apoorva Gupta <apoorvaagupta.info@gmail.com>
2 parents e956cab + 2bee9e0 commit 03f44fe

2 files changed

Lines changed: 126 additions & 3 deletions

File tree

docs/NETWORKING.md

Lines changed: 95 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -601,6 +601,101 @@ sudo systemctl status tailscale-keepalive
601601
tailscale ping ansh-macbookpro-work
602602
```
603603

604+
## TCP Performance Tuning (BBR Congestion Control)
605+
606+
### Problem: TCP Stalls Despite Direct Connection
607+
608+
After solving the DERP relay problem with the keepalive service, Jellyfin streaming still buffered. The direct WireGuard path was established, but TCP throughput was unstable - bursting to 37 Mbps then dropping to 0 Mbps for 2-3 seconds repeatedly.
609+
610+
Symptoms:
611+
612+
- iperf3 TCP tests show thousands of retransmissions per 10-second run
613+
- bandwidth oscillates wildly (0-37 Mbps) instead of holding steady
614+
- Jellyfin buffers despite the connection being direct (not DERP)
615+
- UDP tests at the same target bitrate show near-zero packet loss
616+
617+
### Root Cause: Cubic Congestion Control on a High-Latency Path
618+
619+
The default Linux TCP congestion algorithm (cubic) ramps the send rate until packets drop. On the ~200ms Europe-India path through the Jio router's shallow buffers:
620+
621+
1. Cubic increases the congestion window aggressively
622+
2. The Jio router buffer overflows, dropping packets
623+
3. Cubic detects loss and collapses the window to near-zero
624+
4. The connection stalls for 2-3 seconds
625+
5. Cubic ramps up again, overshoots again - repeat
626+
627+
This produces a sawtooth bandwidth pattern that makes streaming unwatchable, even though the underlying path can sustain 20+ Mbps (proven by UDP tests).
628+
629+
### Fix: BBR Congestion Control
630+
631+
BBR (Bottleneck Bandwidth and Round-trip propagation time) models the path capacity rather than probing until loss. It maintains a steady send rate that fills the pipe without overflowing buffers.
632+
633+
Current configuration on the homelab:
634+
635+
```text
636+
/etc/modules-load.d/bbr.conf:
637+
tcp_bbr
638+
639+
/etc/sysctl.d/99-bbr.conf:
640+
net.ipv4.tcp_congestion_control=bbr
641+
net.core.default_qdisc=fq
642+
```
643+
644+
### Performance Comparison
645+
646+
| Metric | Before (cubic) | After (BBR) |
647+
|--------|---------------|-------------|
648+
| Average throughput (homelab to client) | 8 Mbps | 25 Mbps |
649+
| Lowest interval in 10s test | 0 Mbps (multiple) | 21 Mbps |
650+
| Multi-second stalls | Yes | None |
651+
| Retransmissions per 10s | ~2,400 | ~3,400 (but no stalls) |
652+
653+
BBR still causes retransmissions (the router buffer is shallow), but unlike cubic it does not collapse the send rate when loss is detected. The result is stable, usable throughput.
654+
655+
### Verification
656+
657+
```bash
658+
# Confirm BBR is active
659+
sysctl net.ipv4.tcp_congestion_control
660+
661+
# Expected output:
662+
# net.ipv4.tcp_congestion_control = bbr
663+
664+
# TCP bandwidth test (homelab sends, client receives)
665+
# On homelab:
666+
iperf3 -s -B 100.123.147.108
667+
# On client:
668+
iperf3 -c 100.123.147.108 -t 10 -R
669+
670+
# Expected: stable 20-25 Mbps with no 0 Mbps intervals
671+
```
672+
673+
### Relationship to Keepalive Service
674+
675+
Both components are required:
676+
677+
- The keepalive service maintains a direct WireGuard path (prevents DERP relay fallback)
678+
- BBR ensures the direct path delivers stable throughput for streaming
679+
680+
Without the keepalive, the connection falls back to DERP (~1-3 Mbps). Without BBR, the direct connection has unusable stalls. Neither alone is sufficient.
681+
682+
### Recovery
683+
684+
If BBR causes issues, revert to cubic immediately:
685+
686+
```bash
687+
sudo sysctl -w net.ipv4.tcp_congestion_control=cubic
688+
sudo sysctl -w net.core.default_qdisc=pfifo_fast
689+
```
690+
691+
To make the revert permanent:
692+
693+
```bash
694+
sudo rm /etc/sysctl.d/99-bbr.conf
695+
sudo rm /etc/modules-load.d/bbr.conf
696+
sudo sysctl -p
697+
```
698+
604699
## Related Docs
605700

606701
- [SETUP.md](SETUP.md)

docs/SETUP.md

Lines changed: 31 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -436,7 +436,35 @@ Verify:
436436
sudo ufw status verbose
437437
```
438438

439-
## 7. Create the Shared Docker Network
439+
## 7. Enable BBR Congestion Control
440+
441+
BBR is required for stable streaming throughput over high-latency Tailscale paths. Without it, TCP cubic causes multi-second stalls on international links.
442+
443+
```bash
444+
# Load the BBR kernel module
445+
sudo modprobe tcp_bbr
446+
447+
# Persist module loading at boot
448+
echo 'tcp_bbr' | sudo tee /etc/modules-load.d/bbr.conf
449+
450+
# Enable BBR and fair queueing
451+
sudo sysctl -w net.ipv4.tcp_congestion_control=bbr
452+
sudo sysctl -w net.core.default_qdisc=fq
453+
454+
# Persist sysctl settings
455+
echo -e 'net.ipv4.tcp_congestion_control=bbr\nnet.core.default_qdisc=fq' | sudo tee /etc/sysctl.d/99-bbr.conf
456+
```
457+
458+
Verify:
459+
460+
```bash
461+
sysctl net.ipv4.tcp_congestion_control
462+
# Expected: net.ipv4.tcp_congestion_control = bbr
463+
```
464+
465+
See [NETWORKING.md - TCP Performance Tuning](NETWORKING.md#tcp-performance-tuning-bbr-congestion-control) for full context on why this is needed.
466+
467+
## 8. Create the Shared Docker Network
440468

441469
Reverse-proxied services are expected to join a shared external Docker network named `proxy`.
442470

@@ -453,7 +481,7 @@ docker network ls
453481
docker network inspect proxy
454482
```
455483

456-
## 8. Prepare Persistent Stack Paths
484+
## 9. Prepare Persistent Stack Paths
457485

458486
Before importing stacks into Portainer, ensure the host paths referenced by the compose files exist.
459487

@@ -467,7 +495,7 @@ At minimum, this homelab currently expects paths for:
467495

468496
Do not wait until after stack deployment to create core directories. Missing host paths make troubleshooting much harder and can create permission drift.
469497

470-
## 9. Pre-Deploy Verification Checklist
498+
## 10. Pre-Deploy Verification Checklist
471499

472500
Before deploying the first stack, verify all platform prerequisites:
473501

0 commit comments

Comments
 (0)