Cost Week Report
February 10, 2025
Peak H100 use decreased from 850 to 550.
Model Cost Reductions
| Model | Cost Reduction | New Cost | GPT Cost | Diffusion Cost |
|---|---|---|---|---|
| 13b v4 | 40% | 0.59¢/job | 0.24¢/job | 0.35¢/job |
| 30b v4 | 25% | 1.36¢/job | 1.06¢/job | 0.30¢/job |
| v3.5 | 40% | 0.18¢/job | - | - |
| flux | 30% | 0.09¢/job | - | - |
The Optimizations
GPT
PR #6707: Paged Attention Eviction was finally merged after 5 months, enabling increased batch sizes on GPT. Many workers were running at 50+hz, far exceeding requirements. By adjusting batch sizes to target mid-30hz, we achieved significant throughput improvements.
Diffusion
PR #7296, PR #7406: Starting from 40% GPU utilization, we decoupled the codec from the diffusion worker (thanks Rider!) and moved the upsample engine to a separate process, achieving >90% utilization under load. Worker concurrency was then tuned to maintain load without compromising latency.
Move Fast...
During this week, modal deploy was executed 450 times,
including A/B tests and dev deploys. Every worker underwent
redeployment.
And Break Things
There were 3 cost week related incidents.
- v4 gens were garbled for 10 minutes due to a bad optimization in the diffusion model. I was very confident there was no functional change, but this was very obviously wrong in hindsight.
- All gens were down for 30 minutes when I deployed a worker with a newer modal version. I had previously been testing a new GPU type which required me to update, and I forgot to downgrade afterwards. It turned out modal is not backwards compatible.
- v2 and v3 gens were down for a few hours due to main being broken. Some v4.5 changes to default params broke old models, and thus even simple redeploys broke prod. This went unnoticed for hours since there is almost no traffic in the first place.
My overall takeaways are to think a little bit harder before deploying any change that could have a functional impact, or a worker that hasn't been deployed in a long time.
Next Steps
GPT
30b shows higher efficiency on H200s - consider migration or deprecation. v4.5 will be smaller and potentially run under 25hz. Latency slightly regressed, important to improve this for v4.5.
Diffusion
Flash3 offers potential 10% speedup. Removing CFG could yield 70% improvement. Step reduction or shallow diffusion promises even greater effects. Upsample worker pending engine update.
Codec
We are using more A10s now for codec decoding. These are not well optimized yet. We need to tune concurrency and train a BF16 model.