Motivation
More GPUs do not automatically mean proportionally faster training.
For data-parallel training, every GPU processes a portion of a batch. Gradients must then be synchronized, and the framework has work to do around each step. Sharding model states can lower memory use, but introduces additional communication and execution overhead.
Meshwork measures those costs instead of relying only on theoretical scaling. The study asks two related questions: how close do DDP and FSDP come to ideal throughput as GPU count increases, and how much extra model capacity does FSDP provide for its runtime cost?
Core finding. On this H100 NVLink system, DDP retained over 92% efficiency at four GPUs for both tested model sizes. FSDP traded some throughput for much lower model-state memory and a higher model-size ceiling.
Experimental design
All reported measurements come from a node with four NVIDIA H100 80GB SXM GPUs connected by full NVLink mesh. The software stack uses PyTorch 2.8 and CUDA 12.8. The study evaluates 125M and 350M parameter GPT-2-style language models, using 8 sequences of 1,024 tokens per GPU. Runs cover one, two, and four GPUs; the key tables below show the one- and four-GPU results.
The benchmark records raw measurements and GPU topology under results/gpu/.
Scaling metric
The repository reports weak-scaling efficiency. Each GPU keeps the same local workload, so the global batch grows with GPU count. Efficiency compares multi-GPU throughput with the single-GPU throughput multiplied by the number of GPUs.
This is not strong scaling. The results do not show how much faster a fixed-size global batch would train when split across additional GPUs.
Training strategies
Distributed Data Parallel
DDP stores a full copy of the model and optimizer state on each GPU. Workers compute gradients on their own data and use NCCL all-reduce to synchronize them. Meshwork uses 25 MB gradient buckets, gradient views, bf16 autocast over fp32 master weights, and fused AdamW.
Fully Sharded Data Parallel
FSDP2 splits parameters, gradients, and optimizer states across GPUs. It gathers weights when layers need them, then reshards them after the forward pass. The experiment wraps individual transformer blocks and a root unit, with bf16 parameter computation, fp32 gradient reduction, and fused AdamW. Sharding reduces memory duplication, but every block has additional orchestration and collective operations.
DDP
FSDP2
FSDP still needs temporary gathered parameters and leaves activations and logits unsharded in this setup.
Measured performance
The measured four-GPU throughput shows the difference between the two strategies. DDP stays close to ideal scaling, while FSDP has a larger per-step cost.
| Metric | Single GPU | DDP ×4 | FSDP ×4 |
|---|---|---|---|
| Step time | 32.5 ms | 35.1 ms | 53.3 ms |
| Throughput | 252K tok/s | 933K tok/s | 615K tok/s |
| Weak-scaling efficiency | 100% | 92.5% | 60.9% |
| Model-state memory / GPU | 1.85 GiB | 1.85 GiB | 0.46 GiB |
| Peak memory / GPU | 10.1 GiB | 10.6 GiB | 8.3 GiB |
| Metric | Single GPU | DDP ×4 | FSDP ×4 |
|---|---|---|---|
| Step time | 69.9 ms | 75.7 ms | 90.2 ms |
| Throughput | 117K tok/s | 433K tok/s | 363K tok/s |
| Weak-scaling efficiency | 100% | 92.4% | 77.6% |
| Model-state memory / GPU | 5.29 GiB | 5.29 GiB | 1.32 GiB |
| Peak memory / GPU | 18.2 GiB | 18.2 GiB | 13.0 GiB |
For 125M, four-GPU DDP runs at 933K tokens per second versus 615K for FSDP. At 350M, the gap narrows: 433K versus 363K. FSDP becomes more efficient as useful computation grows relative to its layer-level overhead.
What limits scaling?
The study reruns four-GPU configurations with gradient synchronization disabled. Comparing those steps with standard training helps separate orchestration costs from exposed synchronization time.
| Configuration | Baseline | Wrapper overhead | Exposed sync | Total |
|---|---|---|---|---|
| DDP · 125M | 32.5 ms | +0.8 ms | +1.8 ms | 35.1 ms |
| FSDP · 125M | 32.5 ms | +15.8 ms | +5.0 ms | 53.3 ms |
| DDP · 350M | 69.9 ms | +0.6 ms | +5.2 ms | 75.7 ms |
| FSDP · 350M | 69.9 ms | +8.8 ms | +11.5 ms | 90.2 ms |
“Exposed sync” is the additional observed step time in this probe. Communication may overlap with computation.
DDP hides much of the communication cost
On four GPUs, NCCL all-reduce bus bandwidth measured 375.5 GB/s. DDP communicates about 500 MB of gradients per step for 125M and about 1.4 GB for 350M, yet only 1.8 to 5.2 milliseconds of synchronization remains exposed. Most collective work overlaps with the backward pass.
FSDP pays for per-layer coordination
At 125M, FSDP adds 15.8 milliseconds of wrapper overhead before exposed synchronization is counted. Even on a single GPU with no inter-device communication, FSDP took 50.5 milliseconds per step versus 32.5 milliseconds for unwrapped training. The cost comes from hooks, gathering and freeing weights, and mixed-precision copies. This cost becomes less dominant at 350M.
The optimizer implementation matters
Early four-GPU FSDP runs used unfused AdamW and took 94.6 milliseconds per step on the 125M model. Switching to fused AdamW reduced this to 53.3 milliseconds, about a 1.8× speedup. Gradient clipping added roughly 5 milliseconds on sharded tensors. In this setup, those implementation choices matter as much as some distributed communication optimizations.
Memory and model capacity
FSDP's main benefit is memory. With four GPUs, the repository reports parameter, gradient, and Adam state memory falling from 16 bytes per parameter on each DDP worker to 4 bytes per parameter per FSDP worker. That is a 75% reduction in model-state memory per GPU.
Peak GPU memory does not fall by the same percentage because activations and the logits buffer are not sharded in this experiment. Peak memory drops from 10.6 to 8.3 GiB at 125M, and from 18.2 to 13.0 GiB at 350M.
The 4.85× ratio compares the largest sizes successfully tested. The FSDP run still had around 20 GiB per GPU free; a larger candidate was not tested, so this is a measured lower bound rather than a hard capacity limit.
These results make FSDP useful when model states are the limiting factor. For the smaller models, DDP gives more throughput. As models approach the device memory ceiling, FSDP can make configurations feasible that DDP cannot fit.
Correctness checks
Performance comparisons are only useful if the distributed runs execute the intended training computation. Meshwork uses a deterministic global-batch sampler. For each training step, every worker derives the same global batch from the seed and step number, then takes its own slice. A single-GPU comparison can therefore process the same total tokens.
In the reported checks, DDP and FSDP on four GPUs finish 60 matched steps at the same loss, 10.85485. In fp32, distributed loss curves agree with the single-device curve within 2 × 10−4 on two and four workers. The repository includes nine tests covering correctness, sharding, and parameter counts.
Why this matters. Matching training steps help confirm that measured throughput differences come from execution strategy, rather than different data or learning objectives. They do not prove equivalence across every architecture, precision mode, or training duration.
Implementation and reproducibility
The repository separates model definition, deterministic data sampling, training strategies, memory accounting, and communication benchmarking. Scripts run the GPU study, summarize results, and generate plots. Raw logs and JSON measurements are included under results/gpu/.
| Component | Purpose |
|---|---|
dlt/model.py | GPT-2-style model configurations and parameter counts |
dlt/data.py | Deterministic global-batch sampler |
dlt/train.py | Single-GPU, DDP, and FSDP training with timing probes |
dlt/memory.py | Live model-state memory measurements |
dlt/commbench.py | NCCL collective bandwidth benchmarks |
tests/ | Loss and sharding checks |
Commands follow the repository README. GPU timing results are hardware-dependent and should not be expected to match on other nodes.
Limitations
Scope of the measurements
- All headline benchmarks use a single four-H100 NVLink node.
- Throughput comparisons center on 125M and 350M GPT-2-style models.
- Reported efficiency uses weak scaling, not fixed-global-batch strong scaling.
- The performance study uses eager mode without
torch.compile.
What remains open
- Other interconnects may expose more synchronization time.
- Larger model sizes may shift the compute and communication balance.
- Activation sharding, optimizer choices, and compilation could change tradeoffs.
- Long-run convergence and multi-node scalability are outside the reported study.
The results support conclusions about this hardware and workload, not a universal ranking of distributed training strategies.
Conclusion
Meshwork shows why distributed training needs to be evaluated on two axes: training speed and the model size that fits in memory. DDP is the throughput winner for the tested 125M and 350M models on H100 NVLink, retaining about 92% weak-scaling efficiency at four GPUs. FSDP incurs more per-layer overhead but lowers model-state memory by 75%, allowing a tested 12.85B model where DDP topped out at 2.65B.
The useful lesson is not that one strategy is always better. DDP makes sense when the model fits comfortably and throughput matters most. FSDP becomes valuable when memory duplication blocks a larger model. Separating wrapper overhead from synchronization cost makes that decision easier to explain and reproduce.
References & source material
[1] Sharma, D. Meshwork, source repository, README, implementation, and recorded GPU benchmarks. github.com/b4batunde/Meshwork.
[2] Meshwork benchmark artifacts, results/gpu/, and experimental scripts, scripts/, in the repository above.