
ZenteiQ reports fourfold inference throughput gain with GKE and llm-d
The AMW Read
The reported throughput gain updates a scientific model developer's operating baseline, but its measured impact is limited to one workload.
ZenteiQ reports fourfold inference throughput gain with GKE and llm-d
ZenteiQ, an Indian Institute of Science spinout building scientific foundation models, reported a 4.07-fold increase in batch inference throughput after deploying GKE Inference Gateway with the open-source llm-d framework. In a Google Cloud customer case study, the company reported an increase from 16,964 to 69,114 tokens per second per 1,000 accelerators. The pipeline generates and evaluates synthetic data for model pretraining.
The result shows how inference routing can affect the economics of foundation model development. ZenteiQ uses a fixed allocation of compute for both data generation and training, so delays can leave training clusters waiting for data. Its previous round-robin routing sent requests to busy servers and scattered prompts with shared instructions across instances. The new setup uses queue depth and cached prompt prefixes to route requests. Google and ZenteiQ also reported a 14-percentage-point increase in prefix cache hit rate. The figures are company-reported results for this workload, not an independent benchmark.
For builders, the case supports testing queue-aware and cache-aware routing when batches contain shared prompt prefixes and uneven request lengths. ZenteiQ says 70% to 80% of each prompt's instruction context was shared. For investors evaluating scientific model labs, a useful question is whether data generation can supply training jobs on schedule. Higher token throughput alone does not establish the quality of the resulting datasets or models.