inference

DeepSeek announced the full open-source of its internal inference framework DeepInfer, and simultaneously open-sourced the production-level token encoding and decoding component deepseek-recipe. DeepInfer integrates high-performance operators and a deeply optimized inference engine, supporting distributed inference, speculative decoding, SWA Bounded Replay, and structured output, compatible with NVIDIA and Huawei Ascend platforms.DeepSeek stated that it has implemented high throughput and low latency large-scale inference services for DeepSeek V4.1 Flash based on DeepInfer, and future inference code and optimizations for new models will continue to be updated here. deepseek-recipe is fully aligned with the processing logic of the DeepSeek official API, aiming to ensure that the community receives an experience consistent with the official deployment. The open-source repository and technical report have been released simultaneously.

Micron Technology Vice President Li Zhe stated at the Korea Semiconductor Exhibition SEDEX 2026 held at COEX in Seoul that the company is developing Edge Inference Memory (EIM) for physical AI. The industry expects this product to enter mass production in the second half of 2028. Micron is researching the use of EIM in LPDDR6 to meet the demand for low-power DRAM in applications such as autonomous driving and humanoid robots, and is in discussions with major customers regarding related products.According to a report by the Science and Technology Innovation Board Daily, EIM is positioned between AI chips and memory, utilizing a method of stacking LPDDR on logic chips to process inference data at high speed, although the computing architecture has not been fully disclosed. Li Zhe mentioned that current robotic AI models have about 1 billion to 2 billion parameters, with low bandwidth requirements; however, as more advanced visual language models emerge, LPDDR alone will not suffice, and HBM will eventually be used for humanoid robots. Micron anticipates that future high-performance humanoid robots will require approximately 200 GB of DRAM and 4 TB of storage, while current service robots require about 12 GB of DRAM and 256 GB of storage.Micron CEO Sanjay Mehrotra stated that the storage capacity of a single humanoid robot is about 10 times that of an ordinary L2+ passenger vehicle. The storage industry may enter a period of sustained demand growth for several decades around 2030, driven by the need to synchronize data from hundreds of sensors and run large models locally for millisecond-level control. Counterpoint Research predicts that global DRAM demand for humanoid robots will increase from approximately 8,300 Tbit this year to about 172,000 Tbit by 2030.

Ethereum co-founder Vitalik Buterin shared that the cost of Bitcoin mining is roughly half derived from capital expenditures on computing power equipment and the other half from electricity consumption; in contrast, the cost of AI inference is almost entirely dominated by capital expenditures on computing power. This difference reflects the distinct cost structures of the two computing businesses. Bitcoin miners need to continuously pay for electricity and determine their profit margins based on mining machine efficiency, coin prices, and network difficulty changes; AI inference, on the other hand, relies more on investments in expensive hardware like GPUs, as well as equipment utilization and computing efficiency.The industrial boundaries of both are also changing. As the demand for AI computing power grows, some Bitcoin mining companies are beginning to shift their existing electricity resources and data center facilities towards AI and high-performance computing businesses, hoping to transition from pure mining to computing infrastructure services. Some mining companies are leveraging previously allocated electricity capacity for mining to enter the AI data center market. However, transformation does not mean that cost pressures disappear: mining operations are more directly exposed to fluctuations in electricity prices and Bitcoin prices, while AI computing businesses must bear risks related to GPU procurement, depreciation, and underutilization. Both types of businesses compete for electricity and computing resources, but ultimately, profitability depends on how these resources are converted into revenue.

Blockworks released a status report on cryptocurrency inference service providers, stating that global computing power investment is substantial and demand still exceeds supply; the market capitalization of liquidity crypto AI is approximately $24.9 billion, lower than Crusoe's $30.9 billion, and no decentralized computing network has disclosed verifiable revenue exceeding tens of millions of dollars.The report compares seven crypto-native inference providers on OpenRouter: Venice, Phala, Darkbloom, io.net, AkashML, NEAR AI, and Chutes. Over the past month, the seven have processed an average of about 211 billion tokens daily, accounting for about 1% of the platform's daily average of 21.9 trillion tokens. Venice processed about 87 billion tokens daily, with the team stating in July that its annual recurring revenue reached $100 million, with OpenRouter contributing less than 10%; Phala averaged about 54.6 billion tokens daily over the past week, up from about 5 billion in July.Chutes is the Bittensor inference subnet SN64, operating approximately 4,350 GPUs, with the team claiming to process 160 billion tokens daily, with OpenRouter channels averaging 5.7 billion. Akash has 283 active GPUs, down from a peak of about 897, with an annual leasing fee of about $3.7 million, and GPU incentives have ended. io.net's demand has nearly doubled over the past month to an average of 14.8 billion tokens daily.

According to official news, the P2P AI inference network Antseed has officially been released, while also disclosing that the nonprofit foundation Antseed Foundation has completed a $2.4 million token round of financing. Spark Capital led the investment, with participation from Collider, DCG, North Island Ventures, Reciprocal Ventures, Relay Capital, and Venice.ai.Antseed aims to apply BitTorrent's peer-to-peer model to the AI inference field, allowing different providers to sell models and Agent services directly. After running Antseed's local router on their computers, users can connect tools like Claude Code, Codex, OpenCode, etc., to the same address. Each call is selected by the router based on price, speed, reputation, and privacy conditions, with fees settled directly in USDC on Base. Providers can run their own local models, fine-tune models, or offer other AI services and set their own prices.The biggest difference between Antseed and OpenRouter is that there is no unified platform responsible for forwarding in between. Buyers connect directly to providers, while Antseed is responsible for discovery, routing, reputation, and payment protocols. Ordinary providers can still see the requests they handle; users can also choose providers verified through TEE, allowing requests to run in a trusted execution environment.Antseed is also continuing to improve verification mechanisms such as model fingerprints, signature responses, and audits to prevent providers from misrepresenting cheaper models as more expensive ones. The code has been open-sourced, and the network is operational, although GitHub is currently addressing compatibility issues with Codex, Claude Desktop, and others.

ARK Invest founder Cathie Wood stated that the cost reductions brought about by innovative platforms such as AI will gradually transmit to broader economic sectors, enhancing productivity and corporate profitability, and driving inflation rates down to levels below most investors' expectations, a phenomenon referred to as "benign deflation."Cathie Wood believes that the current AI inference costs are decreasing by 99.99% annually while maintaining performance levels. Meanwhile, OpenAI's annual revenue run rate has increased from $20 billion to $70 billion, indicating that the decline in costs is driving rapid growth in demand.ARK Invest expects that real GDP growth rates could reach high single digits, although this prediction seems "crazy" to most people; in a stronger real growth environment, interest rates may actually rise.

According to The Information, Apple (AAPL.O) is considering a return to the server market and has been in talks with Nvidia (NVDA.O) about using networking technology.The servers that Apple plans to launch are expected to hit the market in 2029, targeting the artificial intelligence inference sector.

Salesforce and NVIDIA jointly launched KOA, which is the first CRM inference model built on NVIDIA NeMoTron.

Zhipu released the lightweight flagship model GLM-5.3 Flash, aimed at competing with DeepSeek V4 Flash, and utilizes over 100,000 domestic chip clusters to provide inference services. The model has a parameter scale of approximately 300B, about 40% of GLM-5.3, and is roughly on par with DeepSeek V4 Flash; it adopts the MoE architecture, with actual activated parameters around 18B, possesses native multimodal capabilities, supports image and video input, and is the first new model with multimodal capabilities following Zhipu's focus on coding strategy.The comprehensive evaluation score exceeds that of the previous flagship GLM-5.2, is on par with Claude Opus 4.8, and is higher than the relevant scores of DeepSeek V4 Pro official version. The pricing is 0.8 yuan per million tokens for input, 2.8 yuan for output, and 0.23 yuan for cache hits, approximately one-tenth of GLM-5.3, with a half-price discount offered for the first two weeks after launch. The model is fully compatible with domestic computing power, online traffic is handled by domestic chips, and Zhipu claims that the cluster hardware efficiency and unit token cost have reached levels comparable to mainstream NVIDIA GPUs; this model was previously tested anonymously on overseas platforms under the code name Ox Alpha.

Sources from relevant personnel at Zhongke Shuguang reveal that the company will launch a new generation of token acceleration solutions aimed at large model inference, focusing on addressing issues of redundant computation and data scheduling during the inference process.Currently, Zhongke Shuguang's ParaStor distributed storage has been applied to the country's first fully domestically produced 100,000-card AI supercluster Shuguang 8000 (Dengfeng), providing data throughput and elastic scaling support for large-scale computing tasks. The latest report from CCID Consulting shows that Zhongke Shuguang ranks first in four sub-industries: AI, education, embodied intelligence, and autonomous driving.

Cerebras released its fourth-generation AI inference system CS-4 this week, based on the same 5nm WSE-3 wafer, achieving double the performance by doubling the clock frequency and power consumption. A single CS-4 cabinet accommodates 3 wafers (CS-3 has 2), featuring a modular "backpack" design that simplifies manufacturing and deployment, with a TDP of approximately 125 to 135kW. The CS-4 can provide an inference speed of nearly 4000 tokens/second/user, about twice that of the CS-3, and supports decomposed inference with heterogeneous systems such as AMD and AWS Trainium.Cerebras claims that the CS-4 offers about 2000 times the on-chip memory bandwidth of NVIDIA's Rubin (43PB/s), but the 44GB SRAM capacity remains unchanged, and long-context inference still requires multi-wafer stacking. For example, with the DeepSeek V4 Pro (1.6T parameters), approximately 20 systems are needed for a 1M context window, and about 40 systems are required for 256 concurrent users, corresponding to a CAPEX exceeding 20 million USD. Cerebras is collaborating with clients such as OpenAI and plans to achieve approximately double performance improvements each year, aiming for a 20-fold throughput increase by 2027. The "backpack" cabinet design of the CS-4 will continue into the next-generation "Nexus" platform.

Design software company Canva has lowered its annual revenue growth forecast by one-third to 20% due to AI feature delivery costs far exceeding expectations. CEO Melanie Perkins stated that user demand for AI features "significantly exceeded expectations," but the company has decided to postpone the large-scale launch of related products until it optimizes its architecture, reduces unit costs, and improves its business model. Since the release of Canva AI 2.0 in April, the cost per task has decreased by nearly 90%, but the volume of designs generated by AI users is three times that of previous versions, and economic pressures remain.This case reflects a common dilemma in the software industry: the cost of AI inference is breaking the traditional software "zero marginal cost" profit logic. Pitchbook analysts pointed out that Canva and Figma are the "most obvious signals" of this trend. Figma's free cash flow profit margin in the second quarter has dropped from 27% in the first quarter to 14%, and revenue growth in the third quarter is expected to slow from 48% to 36%. Canva previously sold employee shares at a valuation of $42 billion, and the market had anticipated its IPO in 2026, but analysts now believe the IPO may be delayed until next year.

Research institution SemiAnalysis released an analysis stating that SpaceX is expected to build approximately 10GW of AI data center capacity by the end of 2027. If 50% of this is used for inference services, with annual revenue exceeding $10 billion per GW, the annualized revenue could reach $300 billion. SpaceX CEO Elon Musk stated in the first earnings report that a "conservative estimate" suggests an additional 6-8GW will be added in 2027, with the actual figure possibly exceeding 10GW.SemiAnalysis's inference simulator shows that when running on the GB300 cluster at current startup cloud prices (about $3/GPU hour), leading model companies like OpenAI and Anthropic could generate annual inference revenue exceeding $10 billion per GW, with annual costs around $12 billion. Microsoft, with full access to OpenAI models and without bearing training costs, can also capture revenue of the same scale. The analysis points out that Microsoft has signed contracts for 10GW of data centers (total value exceeding $300 billion) since 2026, with a 90-day cancellation clause, significantly reducing signing risks.Regarding SpaceX's construction progress, SemiAnalysis believes Musk will significantly shorten the construction cycle by using onsite gas power generation, bypassing large power transformers, parallel construction, and shortening the debugging process. The Southaven plant in Tennessee expanded from 27 turbines (approximately 495MW) in February to 69 turbines (1.7GW) in July, and the "MiniHard" project can be completed in about 5 months with 450-500MW. However, the 10GW target still faces multiple challenges such as land approvals, gas supply, and equipment delivery. This analysis is based on model simulations, and actual implementation still carries uncertainties.

According to a report by Reuters, AMD announced the acquisition of AI inference chip startup Taalas, with the transaction amount yet to be disclosed.The purpose of this acquisition is to enhance AMD's technological capabilities in the field of artificial intelligence inference chips, and it plans to integrate Taalas's technology into its Instinct GPU accelerator roadmap, further advancing the development of system-level solutions.

Jensen Huang announced on the X platform the launch of Alpamayo 2 Super, a cutting-edge open-source inference model developed by NVIDIA for the autonomous driving field. Jensen Huang stated that Alpamayo 2 Super can not only "see" the environment but also understand and reason about the complex world, thinking before acting, and can serve as the core model for robot taxis, autonomous trucks, shuttles, delivery vehicles, agricultural machinery, and future large-scale mobile robot systems.The model will be based on OpenMDW-1.1 and made available for commercial use, allowing developers and enterprises to inspect, fine-tune, and deploy the model. Jensen Huang mentioned that open models help promote the safety and reliability of autonomous driving systems. He stated that the next wave of artificial intelligence will belong to robotics, and autonomous vehicles will be an important starting point for this trend.

According to official news, Marvell Technology announced the launch of a new generation of memory solution portfolio for AI infrastructure, covering server-level AI storage, rack-level CXL memory expansion and pooling, as well as multi-rack optical interconnect shared memory, aimed at addressing the growing memory capacity and bandwidth bottlenecks in Agentic AI inference processes.Marvell stated that as AI model sizes increase, context windows extend, and KV Cache demand grows, traditional tightly coupled architectures of computing and memory are limiting AI inference efficiency. Through memory disaggregation, memory resources can be made more independent of computing resource expansion, improving GPU utilization and reducing data movement latency. The products released include:Bravera SC6 PCIe 6.0 SSD controller: Designed for AI inference storage scenarios, it helps cloud service providers migrate more KV Cache to high-performance SSDs, enhancing infrastructure efficiency. This product features an architecture compatible with multi-vendor NAND and is expected to begin sampling in the fourth quarter of 2026.Marvell Structera X memory expansion solution: Based on CXL technology, it supports rack-level memory expansion and resource pooling, helping data centers share and allocate memory resources more flexibly, reducing AI infrastructure costs.Marvell Photonic Fabric optical interconnect memory solution: Constructs a shared memory architecture across multiple racks using optical interconnect technology, supporting up to 32TB warm KV Cache offloading and helping AI inference clusters enhance throughput capacity.Marvell stated that the Photonic Fabric solution can achieve a 2 to 3 times increase in Token throughput under existing data center space and power consumption constraints, supporting larger scale models and longer context AI applications.Marvell executive Will Chu stated that AI infrastructure is transitioning from a single server architecture to a system where computing, memory, and connectivity operate in synergy, and in the future, memory needs to expand more independently to enhance resource utilization and Token efficiency.As the demand for AI Agents and large model inference continues to grow, memory capacity, bandwidth, and data transfer efficiency are becoming new focal points of competition in AI infrastructure, following computing power.

According to Jinshi reports, the CEO of Amazon Web Services (AWS) stated that the artificial intelligence business is transitioning from the training phase to the inference phase.

According to reports, three informed sources revealed that the Chinese AI startup DeepSeek is developing its own AI chip. This chip is primarily designed for the inference stage of AI computing, rather than for training new models. This move aims to reduce the company's reliance on chips from Nvidia and Huawei.The report noted that the chip development plan began about a year ago and is still in the early stages. DeepSeek is discussing with external partners such as chip design firms, foundries, and storage companies, and has increased its recruitment of chip design engineers through private channels in recent months. Additionally, following the news, Nvidia's pre-market stock price fell slightly by about 1.6%.Analysts believe that if the development is successful, it will mark a significant strategic shift for DeepSeek, aligning with the global trend of leading AI developers seeking hardware autonomy, and may also pose a challenge to Huawei's position in the domestic AI chip market. However, the project still faces challenges such as long design cycles, high funding requirements, and manufacturing barriers like U.S. export restrictions on advanced processes and high-bandwidth memory (HBM). This chip development initiative also aligns with recent reports about DeepSeek's plan to raise $7 billion in its first round of financing.