You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[Feature Request] support tensor parallelism for multi gpu setup #156
As of today (v0.1.25), strata supports only layer split. I wonder if tensor split (tensor parallelism) can be added for homogeneous GPUs. This could improve the computation with a small cost of VRAM.
Thanks for the suggestion. For now multi-GPU means the layer split plus the secondary expert caches. Decode is mostly limited by the experts, so tensor parallelism of the dense part would gain less here than on a dense model. We'll keep this open as a feature request.
Closing as not planned for now: decode is limited by the experts, so tensor parallelism of the dense part gains little here. Multi-GPU work continues in the layer split (#216). It stays on our backlog.
Just for completenes' sake: Would tensor parallel decode allow for asymmetrical GPU-setups (1 x Fast GPU, 1 x rather slow) to provide a benefit over just using only the fast one?
Just for completenes' sake: Would tensor parallel decode allow for asymmetrical GPU-setups (1 x Fast GPU, 1 x rather slow) to provide a benefit over just using only the fast one?
It will run but at the slower card rate. So TP is best for homogenous cards.
5. > Just for completenes' sake: Would tensor parallel decode allow for asymmetrical GPU-setups (1 x Fast GPU, 1 x rather slow) to provide a benefit over just using only the fast one?
It will run but at the slower card rate. So TP is best for homogenous cards.
Hmmm... I thought that was exactly what happens when you choose layer split. At least with dense models, my asymmetrical GPUs worked "meh" in layer split but fairly well in tensor split. I'm not very knowledgeable about these things, though. Thanks!
5. > Just for completenes' sake: Would tensor parallel decode allow for asymmetrical GPU-setups (1 x Fast GPU, 1 x rather slow) to provide a benefit over just using only the fast one?
It will run but at the slower card rate. So TP is best for homogenous cards.
Hmmm... I thought that was exactly what happens when you choose layer split. At least with dense models, my asymmetrical GPUs worked "meh" in layer split but fairly well in tensor split. I'm not very knowledgeable about these things, though. Thanks!
Layer split takes turns to run on cards like a relay race so when it's your faster card’s turn (to handle the assigned layers), it runs fast. TP is running multiple cards in sync. So the slowest one limits the total speed. It's like two horses run together.
9. @Niko1221 I don't entirely understand. How are the experts a limiting factor?
In my understanding, qwen3.8 flash next only has 6B parameters active. And most of the time (especially when you have VRAM hit rate > 75% on one of your cards), most computation will be done on a single gpu. The other gpus will not have to run much computing. It might seem that using tp so all gpu will have to work at the same time will increase prefill and decode speed. But that's not the case. TP uses allreduce to sync the gpus after once per layer on every token (qwen3.8 flash next, may differ per models) and allreduce is a heavy computation. So on every token, you are essentially adding more computation. With dense models, this would make sense since the computing allreduce will take much shorter time instead of each gpu waiting for the other on every token. But if you're using an moe model with very small active parameters and most of the active experts are sitting on one gpu, the gpus will be faster taking turns and using allreduce will only bring additional computation which will essentially slow it down. Also, gpu to gpu communication is bounded by pcie (32-64GB/s maybe even less if you are using old pcie gen) unless you have nvlink (160GB/s - 3TB/s and is not cheap). TP will make multiple communications (48 times in the case of qwen3.8 flash next) between gpus per token and all reduce will be computed for each communication while layer split will only do one communication per gpu on every token. So tp will make it slower not faster. I might be wrong on some details but I think that's the gist of it.
Activity
Niko1221 commentedon Sep 29, 2026
Thanks for the suggestion. For now multi-GPU means the layer split plus the secondary expert caches. Decode is mostly limited by the experts, so tensor parallelism of the dense part would gain less here than on a dense model. We'll keep this open as a feature request.
Niko1221 commentedon Sep 30, 2026
Closing as not planned for now: decode is limited by the experts, so tensor parallelism of the dense part gains little here. Multi-GPU work continues in the layer split (#216). It stays on our backlog.
MaKla89 commentedon Oct 17, 2026
Just for completenes' sake: Would tensor parallel decode allow for asymmetrical GPU-setups (1 x Fast GPU, 1 x rather slow) to provide a benefit over just using only the fast one?
megakilo commentedon Oct 17, 2026
It will run but at the slower card rate. So TP is best for homogenous cards.
MaKla89 commentedon Oct 17, 2026
Hmmm... I thought that was exactly what happens when you choose layer split. At least with dense models, my asymmetrical GPUs worked "meh" in layer split but fairly well in tensor split. I'm not very knowledgeable about these things, though. Thanks!
megakilo commentedon Oct 17, 2026
Layer split takes turns to run on cards like a relay race so when it's your faster card’s turn (to handle the assigned layers), it runs fast. TP is running multiple cards in sync. So the slowest one limits the total speed. It's like two horses run together.
chriskerley78910 commentedon Oct 26, 2026
@Niko1221 I don't entirely understand. How are the experts a limiting factor?
jackieveev commentedon Oct 27, 2026
Same. Its a pity that strata not support tensor split, waiting for it!
xXRex45Xx commentedon Oct 27, 2026
In my understanding, qwen3.8 flash next only has 6B parameters active. And most of the time (especially when you have VRAM hit rate > 75% on one of your cards), most computation will be done on a single gpu. The other gpus will not have to run much computing. It might seem that using tp so all gpu will have to work at the same time will increase prefill and decode speed. But that's not the case. TP uses allreduce to sync the gpus after once per layer on every token (qwen3.8 flash next, may differ per models) and allreduce is a heavy computation. So on every token, you are essentially adding more computation. With dense models, this would make sense since the computing allreduce will take much shorter time instead of each gpu waiting for the other on every token. But if you're using an moe model with very small active parameters and most of the active experts are sitting on one gpu, the gpus will be faster taking turns and using allreduce will only bring additional computation which will essentially slow it down. Also, gpu to gpu communication is bounded by pcie (32-64GB/s maybe even less if you are using old pcie gen) unless you have nvlink (160GB/s - 3TB/s and is not cheap). TP will make multiple communications (48 times in the case of qwen3.8 flash next) between gpus per token and all reduce will be computed for each communication while layer split will only do one communication per gpu on every token. So tp will make it slower not faster. I might be wrong on some details but I think that's the gist of it.