DeepSeek open-sources DeepInfer, the software it uses to run AI models at scale
In short
China's DeepSeek said on the 20th that it is releasing software it uses to run its AI models at scale. The tool, called DeepInfer, currently supports the V4.1-Flash model. Bloomberg noted there is no independent benchmark of its efficiency gains yet.
Read the full story 2 min read
DeepSeek, the Chinese AI company, said on the 20th that it would release software that raises the efficiency of inference for its latest AI model, Nikkei reported from Guangzhou. Nikkei said the software makes downloaded models easier to use and that releasing such software is rare. [ 2 ]
Bloomberg, in a report carried by TBS NEWS DIG, said the software is called DeepInfer and is what DeepSeek uses to run its models at scale. The company, which is headquartered in Hangzhou, wrote on X on the 20th that it was releasing the software used in production so that the community can reduce computing costs and energy use and make AI more affordable. DeepInfer currently supports the V4.1-Flash model, with future systems to be supported in turn, according to the report. [ 3 ]
Gigazine reported that DeepInfer was developed on the basis of vLLM, with ideas from SGLang and TensorRT LLM also incorporated. The open-source version collects the core functions of the system DeepSeek uses in its AI services. It supports V4.1-Flash and offers optimised inference and an acceleration technique called DSpark, but the KV cache system is not implemented. It is published on GitHub under the Apache License 2.0. [ 1 ]
Gigazine also reported that DeepSeek open-sourced deepseek-recipe, a collection of libraries that convert API requests into a form suited to DeepSeek models and convert model output into the API response format. This allows third-party API providers to reproduce the behaviour of DeepSeek's official API. It is published under the MIT License. [ 1 ]
Bloomberg said the release shows how DeepSeek keeps down the cost of inference, which AI companies usually guard closely, and could let cloud and hosting providers adopt such cost-cutting methods themselves. The chief executive of Requesty, a startup working on AI gateway metrics and routing, said the disclosure lowers the barrier to hosting DeepSeek models but that computing capacity and expertise are still required. Bloomberg added that there is currently no independent benchmark evaluation of DeepInfer's efficiency gains. [ 3 ]
Why it matters
Bloomberg said the release could let corporate users and hosting providers run DeepSeek models at lower cost. The documents give no measured results, so the size of any saving is not yet known.
Key facts
- DeepSeek announced on the 20th that it would release software to raise the efficiency of inference for its latest AI model. [ 2 ]
- The software is called DeepInfer and is used in DeepSeek's own AI services; the open-source version contains its core functions. [ 1 , 3 ]
- DeepInfer currently supports DeepSeek's V4.1-Flash model, with support for future systems planned. [ 1 , 3 ]
- DeepInfer is published on GitHub under the Apache License 2.0; the open-source version does not implement the KV cache system. [ 1 ]
- DeepSeek also open-sourced deepseek-recipe under the MIT License, a collection of libraries that lets third-party API providers reproduce the behaviour of DeepSeek's official API. [ 1 ]
- Nikkei said releasing such software is rare. [ 2 ]
Confirmed by several sources
Still unclear
- How much DeepInfer improves efficiency or reduces cost. Bloomberg said there is currently no independent benchmark evaluation of DeepInfer's efficiency gains.
- Whether the open-source version matches the version DeepSeek runs in production. Gigazine said the open-source version contains the core functions and lacks the KV cache system; no other document describes the differences.