Skip to main content
Back to News
ZenteiQ reports fourfold inference throughput gain with GKE and llm-d
Technology
2 min read
IN

ZenteiQ reports fourfold inference throughput gain with GKE and llm-d

The AMW Read

The reported throughput gain updates a scientific model developer's operating baseline, but its measured impact is limited to one workload.
NoveltySignificance
Foundation Models · Player Map
Zenteiq.ai
Zenteiq.ai

Foundation Models / LLMs

View Company Profile

ZenteiQ reports fourfold inference throughput gain with GKE and llm-d

ZenteiQ, an Indian Institute of Science spinout building scientific foundation models, reported a 4.07-fold increase in batch inference throughput after deploying GKE Inference Gateway with the open-source llm-d framework. In a Google Cloud customer case study, the company reported an increase from 16,964 to 69,114 tokens per second per 1,000 accelerators. The pipeline generates and evaluates synthetic data for model pretraining.

The result shows how inference routing can affect the economics of foundation model development. ZenteiQ uses a fixed allocation of compute for both data generation and training, so delays can leave training clusters waiting for data. Its previous round-robin routing sent requests to busy servers and scattered prompts with shared instructions across instances. The new setup uses queue depth and cached prompt prefixes to route requests. Google and ZenteiQ also reported a 14-percentage-point increase in prefix cache hit rate. The figures are company-reported results for this workload, not an independent benchmark.

For builders, the case supports testing queue-aware and cache-aware routing when batches contain shared prompt prefixes and uneven request lengths. ZenteiQ says 70% to 80% of each prompt's instruction context was shared. For investors evaluating scientific model labs, a useful question is whether data generation can supply training jobs on schedule. Higher token throughput alone does not establish the quality of the resulting datasets or models.

#FoundationModels #AIInfrastructure #Inference #Kubernetes #ScientificAI

#ZenteiQ#batch inference#GKE Inference Gateway#llm-d#related:Google

How This Connects

Based on Foundation Models · Player Map

  1. 1d agoOpenAI reports chip design gains on Jalapeno and AMD hardwareOpenAI
  2. 1w agoDeepSeek Reportedly Raises $15 Billion With Tencent and CATL ParticipationDeepSeek
  3. 3w agoCohere completes initial close of Series E targeting up to CAD 7 billionCohere
  4. 3w agoCohere secures up to C$1.6 billion from Canada with conditions on jobs and headquartersCohere
  5. 3w agoCohere completes first close of funding round targeting up to C$7 billionCohere
  6. 3w agoZenteiQ reports fourfold inference throughput gain with GKE and llm-d · THIS ARTICLE

More news from Zenteiq.ai

Stay updated with the latest news and announcements from Zenteiq.ai.

View all Zenteiq.ai news

Discover AI Startups

Explore 5,000+ AI companies with VC-grade analysis, funding data, and investment insights.

Explore Dashboard