DeepSeek Opens Its Production Inference Engine, With Huawei Hardware Gaining Equal Footing to Nvidia

The Chinese AI company has made public the inference engine that operates its production models, reversing an earlier stance that such a release was not yet feasible. Branded DeepInfer, this framework enables developers to deploy DeepSeek's V4.1 Flash model across their own systems while retaining many of the performance enhancements DeepSeek applies within its own service. The company has signaled that support for upcoming DeepSeek models will follow.

DeepSeek has long permitted developers to obtain and run its model weights through external tools like vLLM. However, the infrastructure layer—the software orchestrating requests, coordinating computation across processors, and producing outputs—represents an entirely separate challenge. This layer fundamentally shapes latency, throughput, and resource consumption. By releasing DeepInfer, DeepSeek essentially opens its own solution to these operational problems.

We're sharing our production stack to help the community lower compute costs, reduce energy use, and make AI more affordable.

DeepSeek

Making AI more affordable

On Tuesday, DeepSeek announced via X that it was releasing its internal inference technology to external developers as a means of reducing the financial burden of operating large language models. The framework carries an Apache 2.0 license and incorporates optimized computational kernels and communication protocols. These include DeepSeek's previously released FlashMLA, DeepGEMM, and DeepEP components, together with systems for spreading inference workloads across multiple processors.

The package incorporates speculative decoding, which accelerates token generation by forecasting and confirming multiple tokens simultaneously, plus asynchronous scheduling that minimizes idle processor time. Developers can also leverage structured outputs, a capability that lets them define the format of model responses.

https://x.com/deepseek_ai/status/2112476496283775135?ref_src=twsrc%5Etfw

Accompanying DeepInfer is deepseek-recipe, a supplementary library that transforms API requests into model inputs and converts model outputs into API responses. Offered in both Rust and Python implementations, this tool enables developers to replicate the behavior of DeepSeek's public API when running independent deployments. The library debuted quietly in September, with an updated release published Tuesday.

The announcement arrives shortly after DeepSeek unveiled DeepSeek-V4.1-Flash-1017, an updated iteration of V4.1 Flash emphasizing gains in code generation and computational efficiency. That same period saw DeepSeek make its proprietary web search technology accessible to developers through new Search and Fetch APIs.

Why DeepSeek held back in 2025

The current DeepInfer release represents a more expansive disclosure than DeepSeek's earlier intentions. In April 2025, the company published a strategy document detailing its inference engine plans, noting that while it had explored releasing the complete system, it had opted instead to contribute isolated modules and enhancements to established projects.

DeepSeek cited multiple barriers at that time. The inference software had evolved from an initial vLLM fork into a highly specialized platform, and the company doubted whether its modest research group possessed the capacity to steward an additional major open-source initiative. Practical obstacles also existed in adapting the software for environments outside DeepSeek's own data centers.

The engine is tightly coupled with our internal infrastructure, including cluster management tools, making it impractical for public deployment without significant modifications.

DeepSeek, April 2025

What circumstances have shifted? DeepSeek has not publicly detailed how it resolved those earlier constraints, though the character of this release hints at answers. According to DeepInfer's documentation, the public version presently handles offline inference and encompasses "the core features needed for serving at scale." Nonetheless, several elements essential for running a complete production environment remain absent. Missing components include a KV cache management layer, infrastructure for distributing inference stages across separate machines, and load balancing across multiple instances.

In essence, DeepSeek has made accessible the central engine that executes its models while leaving developers responsible for constructing auxiliary infrastructure needed to operate a full service. This more limited scope may clarify how DeepSeek managed to release its technology despite the obstacles it articulated last year.

Nvidia and Huawei, side by side

Beyond the inference engine itself, the hardware compatibility announced from launch merits particular attention. Nvidia appears in the supported list, though limited to its SM100 architecture, which powers Blackwell data center GPUs. Equally notable is the inclusion of Huawei's Ascend 950 NPU series, positioning the Chinese semiconductor manufacturer alongside Nvidia among DeepInfer's compatible platforms. Notably absent are older Nvidia designs, including Hopper.

This decision connects to DeepSeek's public roadmap from 2025. Although the company was reluctant to open its full inference engine at that moment, it pledged to collaborate with chipmakers in advance of new model releases, aiming to make its models available across diverse chip architectures immediately upon launch.

U.S. export controls have restricted Chinese firms' access to Nvidia's cutting-edge processors, adding context to DeepSeek's deepening partnership with Huawei. In September, Bloomberg disclosed that DeepSeek intended to deploy at least 160,000 Huawei Ascend 950DT chips at a facility in Inner Mongolia dedicated to running its models. While DeepSeek has principally depended on Nvidia for model training, The Information reported in September that the company plans to expand its deployment of Huawei chips for training, with deliveries anticipated in the fourth quarter. Later that month, Reuters reported that DeepSeek and Huawei were developing software utilities designed to streamline programming for Ascend processors, underscoring DeepSeek's commitment to Chinese AI hardware.

The Huawei support has already generated discussion in developer communities. On Reddit, commenters highlighted the dual hardware support as a significant aspect of the announcement, while others speculated that DeepSeek's background in quantitative finance might explain its optimization prowess. One user remarked, "Once again Deepseek shows how good high-frequency traders are at optimizing performance… On Ascend NPUs nonetheless!!" Another user made a jest about Anthropic's Claude Opus 5.5, referencing reports that requests involving kernel programming for Huawei's AI chips could activate safeguards that route certain advanced development work to an older model version. "Don't let Opus 5.5 see this! It will delete the Ascend support," the user wrote.

Feature parity between the Nvidia and Huawei implementations does not exist. Batch-invariant inference and deterministic sampling—features that guarantee reproducible outputs under defined parameters—are restricted to Nvidia hardware. DeepSeek's published performance benchmarks employ Nvidia GPUs, utilizing four SM100 units for prefill and 64 for decoding, leaving the comparative performance on Huawei processors as an open question.

For developers lacking access to either processor family, the immediate utility may stem from examining the released code and applying its optimization strategies to alternative inference systems. DeepSeek has nonetheless provided the broader open-source ecosystem with substantially more of its inference technology to explore and build upon.