DeepSeek open-sources DeepInfer: in-house inference and Huawei alongside Nvidia

DeepSeek has taken a step that many IT teams had been waiting for: it has open-sourced DeepInfer, the inference engine it uses in production to serve its models. This is not just the model weights, which were already available, but the layer that manages requests, distributes work across chips, and generates responses. In other words, the part that makes the difference between being able to run a model and running it efficiently.

The announcement, shared by the company on X, comes with an Apache 2.0 license and a stated goal: to reduce compute costs, lower energy consumption, and make AI more affordable. For a Spanish SMB considering deploying models on its own infrastructure, the news deserves attention, albeit with important nuances that should not be overlooked.

What DeepInfer actually includes

The framework is designed to run the DeepSeek V4.1 Flash model on its own infrastructure, with many of the optimizations the company applies in its API. There are also plans to support future models. Inside the package we find optimized compute kernels and communication libraries that DeepSeek had already released separately, such as FlashMLA, DeepGEMM, and DeepEP, as well as mechanisms to distribute inference across multiple chips.

Two techniques stand out for their practical impact:

• Speculative decoding: speeds up response generation by predicting and verifying multiple tokens at once.
• Asynchronous scheduling: reduces the time processors spend waiting for new work.

In addition, the framework supports structured outputs, which makes it possible to specify how the model's response should be formatted. Alongside DeepInfer, DeepSeek releases deepseek-recipe, a complementary library, available in Rust and Python, that manages how API requests are converted into model inputs and how the output is processed. Its purpose is for a self-hosted deployment to behave similarly to the official API.

What is still missing for production

It is worth being honest: DeepInfer is not yet a turnkey solution. The documentation indicates that the publicly available software supports offline inference and the essential functions for serving at scale, but key pieces for operating a complete service are missing. Among them, a KV cache management system, infrastructure to separate the different inference phases across machines, and load balancing between replicas.

In other words: DeepSeek opens up much of the engine, but leaves it to the developer to provide the surrounding infrastructure. This partly explains how the company has overcome the difficulties it itself pointed out in 2025, when it argued that its engine was so tightly coupled to its internal infrastructure that it was not practical to release it as is.

Huawei alongside Nvidia: the fine print matters

The perhaps most interesting detail is not the engine itself, but the hardware supported from day one. Nvidia is present, as expected, but with a striking restriction: only the SM100 architecture, the one used in Blackwell data center GPUs. Earlier Nvidia architectures, including Hopper, are conspicuously absent.

The other compatible platform is Huawei's Ascend 950 series, which places the Chinese manufacturer on the same level as Nvidia in DeepSeek's list of supported hardware. This move has a geopolitical and technical reading: it strengthens the alternative compute ecosystem and forces IT teams to ask themselves what hardware they will actually have available when they want to deploy this type of model.

What it means for an SMB

For most SMBs, the temptation to set up DeepInfer on their own server is premature. Deployment requires specific hardware, knowledge of distributed inference, and the ability to build the missing pieces. It is not a weekend project, nor something that can be solved with a mid-range GPU.

That said, there are three conclusions worth keeping in mind:

1. The cost of local AI keeps falling. The fact that a player like DeepSeek is releasing its production stack indicates that inference efficiency is becoming an open field, not an industrial secret. This pushes cloud service prices down and broadens the options.
2. Hardware dependency is real. Before considering any local deployment, check which architectures the software you want to use supports. Buying GPUs without checking compatibility is an expensive way to waste time.
3. Data sovereignty gains ground. For sectors with strict requirements, being able to run models on their own infrastructure, even with nuances, opens conversations that were not even considered before.

At ForgeNEX we recommend a prudent approach: follow the evolution of DeepInfer, test it in lab environments if the team has a technical profile, but for now do not replace a managed API with a self-hosted deployment without a total cost analysis that includes hardware, people, and maintenance. Accessible AI is not just a matter of licenses; it is a matter of knowing what you are buying.

If your SMB is evaluating how to integrate AI without blowing up the technical bill, you may be interested in reviewing our analysis on how AI lowers the barrier to building software or the article on whether local AI is worth it for SMBs. The infrastructure decision is rarely made in a vacuum.

Source: The New Stack. Analysis and adaptation: ForgeNEX.