DeepSeek releases DeepInfer's offline inference stack under Apache 2.0
The AMW Read
The release repeats the DeepInfer event already covered on October 21, while its production-derived inference code has segment-level relevance tempered by single-model support and omitted serving components.
DeepSeek releases DeepInfer's offline inference stack under Apache 2.0
DeepSeek released DeepInfer on GitHub on October 20, opening the inference framework behind its large-scale production services. The release covers offline inference and supports only DeepSeek-V4.1-Flash, downloading the DeepSeek-V4.1-Flash-1017 checkpoint from Hugging Face on first execution. It supports NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs and requires a source build. DeepSeek published the code under Apache License 2.0.
The release puts a foundation-model developer's execution technology into the AI infrastructure market, where scheduling, parallelism and optimized kernels shape serving efficiency. DeepInfer includes attention data and context parallelism, mixture-of-experts parallelism with load balancing, asynchronous batch scheduling, deterministic sampling and structured output. Initially built mainly on vLLM, with references to SGLang and TensorRT-LLM, it adds another implementation to the open-source inference ecosystem. The scope matters: production use inside DeepSeek does not establish that the published package is a complete serving system.
Builders should assess the missing service components before committing engineering resources. The release excludes KV-cache infrastructure, separation of VE, prefill and decode, and load balancing across prefill and decode data-parallel replicas. Performance measurements used four SM100 GPUs for prefill and 64 for decode, with reproduction scripts included. Those conditions give teams a concrete evaluation setup, but do not establish equivalent performance on other deployments or on Ascend hardware. Its immediate utility therefore depends on matching the supported model and hardware and supplying the omitted serving components.
