training

Dwarkesh Patel and Jerry Han published research on September 8, 2026, breaking down the contributions of data and model improvements in pre-training progress from 2019 to 2025. The study was conducted on a smaller scale, focusing on pre-training, corresponding to the publicly released model recipes and publicly available data corpora for each year, and was trained at a computational scale of up to 1e19 FLOPs.The results show that under the 1e19 FLOPs computational budget, the computational efficiency gain from data improvements is 12.0 times, while model improvements yield 3.7 times, with the data-side gain being approximately 3.24 times that of the model side. The gains from data and model improvements are largely independent and interact very little, with the additive effect of both explaining 88% of the variance in OLMES scores under a linear model.On the model side, the evolution from GPT-2 to OLMo-2 encompasses optimizers, positional encoding, normalization, activation functions, and initialization, among others. On the data side, the corpus evolved from approximately 9 billion tokens of OpenWebText in 2019 to larger and more finely filtered corpora like UltraFineWeb by 2025.

Pequity Research cited FundaAI stating that the training computing power used by GPT-6 is approximately 10 times that of GPT-5.FundaAI indicated that based on industry discussions, the total computing power consumed by experiments, reinforcement learning, synthetic data generation, and supporting infrastructure could be several times that of the main pre-training run, potentially reaching up to ten times. Once TPU 8t and Vera Rubin are shipped on a large scale, the next round of pre-training acceleration can begin, with large-scale interconnection and optics benefiting the most during this cycle.

According to TechCrunch, citing multiple informed sources, less than three months after emerging from stealth mode, the startup XDOF, which specializes in collecting real-world remote control data for training general-purpose robots, is in the late stages of negotiations for a Series B funding round, with an expected valuation of around $1.2 billion. This round of financing is led by 8VC.XDOF was co-founded in 2024 by researchers Philipp Wu (CEO) and Fred Shentu (CTO) from the University of California, Berkeley. The company previously completed a $70 million Series A funding round, with investors including Thrive Capital, Andreessen Horowitz, Lux, and Spark Capital. XDOF did not originally plan to raise funds again so quickly after completing this round, but informed sources say the company's business is growing rapidly, with annual revenue nearing $50 million, thus attracting several venture capital firms to proactively reach out and push for a new funding round.TechCrunch was unable to ascertain the specific amount of this funding round and could not confirm whether the aforementioned $1.2 billion valuation includes the new financing. Informed sources indicated that the transaction terms have not yet been finalized and may still change.

According to CriptoNoticias, the Argentine Public Prosecutor's Office (MPF) conducted a special training course titled "Virtual Assets: Financial Analysis, Tracking, Detection, and Seizure" for internal staff, officials, and judges on August 19 and September 2, 2026, via the Zoom platform, with anti-money laundering specialist lawyer Carmen Chena as the speaker.The course covered topics such as digital wallet asset analysis, domestic and international legal frameworks, the cryptocurrency ecosystem, and practical case studies on preservation measures. Previously, the Argentine judicial system had accumulated rich experience in cryptocurrency-related cases, including the freezing of 3 million USDT in December 2024 and the ruling in 2022 for Binance to return stolen BTC.

According to market news, AI training data startup AfterQuery has completed a new round of financing, with a valuation reaching $3.2 billion. Just five months after completing a $30 million Series A round at a valuation of $300 million in April, the valuation has increased more than tenfold.According to Y Combinator partner Gustaf Alströmer, this is the fastest startup in YC history to go from founding to becoming a unicorn. The founders of AfterQuery are only 22 and 23 years old and just participated in the YC Winter 2025 batch 18 months ago. The company announced in April that its annualized revenue has reached $100 million, with clients including NVIDIA, Legora, and the South Korean AI lab Motif Technologies. AfterQuery focuses on training models and agents to complete tasks like professionals rather than just ensuring the accuracy of answers.

Anthropic stated that the company has temporarily paused certain AI training and cybersecurity assessments. The company disclosed the relevant adjustments in a blog post, explaining the measures taken in response to unauthorized actions by its AI agents earlier this year. OpenAI previously indicated that it had paused some model development work due to safety concerns. Anthropic also confirmed that it has taken similar measures and emphasized the need for broader coordination on the pace of frontier AI development.Anthropic stated that after disclosing three related incidents in July, the company paused external cybersecurity assessments for pre-release models and also briefly suspended internal testing of pre-release models. Following the incidents, the company also paused high-risk reinforcement learning environments in pre-release models for several weeks. Anthropic noted that most reinforcement learning work has resumed, but some high-risk environments remain on hold, awaiting human review or updates to monitoring tools.

According to The Verge, Sony Music Publishing and Warner Chappell Music have filed a lawsuit against Anthropic in the U.S. District Court for the Northern District of California, seeking damages for "tens of thousands" of copyrighted works. They accuse Anthropic of illegally downloading, scraping, and distributing copyrighted works on a large scale to develop and train the Claude series of AI models. The plaintiffs are seeking up to $150,000 in damages for each work and an additional claim of up to $25,000 for each instance of removing identifiable copyright data.In addition to suing Anthropic, the lawsuit also names co-founders Dario Amodei and Benjamin Mann as individual defendants. The lawsuit alleges that Mann used BitTorrent to download over 5 million pirated books, and that employees downloaded at least 2 million from Pirate Library Mirror. The complaint also claims that Anthropic scraped lyrics from websites such as MusixMatch and LyricFind. Recently, Anthropic reached a $1.5 billion settlement with the publishing industry over another lawsuit, while also facing multiple lawsuits from Universal Music Group, Concord, and ABKCO, as well as lawsuits filed by BMG and Round Hill Music.

According to TIME, OpenAI CEO Sam Altman recently stated in an interview that the company has previewed the upcoming cutting-edge model series Astra to key clients. In the demonstration, 16 AI agents can collaboratively break down mathematical problems and assemble proofs, and Astra can operate computer software across applications at superhuman speeds. Altman mentioned that Astra will support "persistent agents" capable of performing long-term tasks and is expected to be the first model that can invent new things in a meaningful way, possessing characteristics of AGI.Over the past year, OpenAI has fallen behind expectations in product direction and pre-training research, being surpassed by Anthropic in programming products, annual revenue, and valuation. The company has experienced multiple executive departures and is facing challenges such as several product liability lawsuits and legal disputes with Apple and Musk. OpenAI's current valuation is nearly $1 trillion, with ChatGPT having over 1 billion monthly active users.Recently, OpenAI disclosed a security incident: an unreleased agent escaped the sandbox and attacked Hugging Face. Following this, the research team froze some experiments, enhanced monitoring, and paused the training of an unreleased model expected to bring the greatest capability leap until new safety measures are in place. Altman emphasized that "ensuring AI safety is more important than the growth momentum of any company," and the company will slow its pace and allocate resources to safety and alignment teams. Chief Research Officer Mark Chen estimated that the company is about 80% complete in reaching AGI, and Altman stated that the internal system may be referred to as AGI by the end of the year.

According to X user synthwavedd, OpenAI has recently completed the next phase of pre-training, internally codenamed "Bel," which is the successor model to the previous "Doug." This model has a total parameter count exceeding 10T, comparable in scale to GPT-4.5. It is expected that after further reinforcement learning, it will become the foundational model for Astra and GPT-6, and is anticipated to serve as the base model approaching the AGI threshold in the post-GPT-6 era.Reports indicate that this development has caused concern for Anthropic. OpenAI believes that Anthropic currently lacks an effective plan to respond to the public release of Astra; insiders claim that this assessment is largely accurate, primarily due to limited computing power. Anthropic is preparing for the possibility that OpenAI will dominate for the remainder of this year, but it is expected to regain a lead early next year.

According to TrendForce, at Hot Chips 2026, Micron emphasized that the progress of AI computing power is faster than storage improvements, and the challenge of the "storage wall" is becoming increasingly prominent. Micron Fellow Raghu Sriramaneni pointed out that the computing power of AI accelerators increases approximately threefold every two years, while the bandwidth of HBM increases by less than double during the same period, and the gap may further widen. The deepening reliance on HBM also brings reliability issues; Micron cited Meta Llama 3 data indicating that HBM failures account for about 17% of unexpected training interruptions. New designs that blur the boundaries between computing and storage may help break through the bottleneck.The expansion of HBM also brings area and supply pressures: the latest generation of packaging integrates two GPUs with eight 12-layer HBM4 stacks, with storage accounting for about 90% of the semiconductor area, more than eight times the area occupied by GPUs; achieving equivalent capacity with HBM requires about three times the wafers of standard DDR5 DRAM. Thermal management has also become a key constraint, with solutions such as liquid cooling and ultra-thin chips being explored.Micron is advancing innovations in interconnect, packaging, and cooling, including SerDes and die-to-die PHY optimized for storage, larger size SiP and advanced packaging with glass substrates, as well as liquid cooling and hybrid bonding, and is developing fusion bonding to reduce thermal resistance and enhance data throughput.

According to LatePost, ByteDance is discussing a large model with training parameters exceeding 50 trillion, surpassing Alibaba's Qwen 3.8-Max (24 trillion) and Moonlight K3 (28 trillion), making it the largest known plan in the country so far. This plan is still in its early stages and does not guarantee a final release. The new model is intended to be led by Xiang Liang, head of Seed Foundation, in collaboration with Shen Ke, who is responsible for the pre-training data of large language models. Seed is reorganizing, dividing responsibilities, and allocating resources based on this.Two weeks ago, ByteDance founder Zhang Yiming held a company-wide meeting with Seed head Wu Yonghui. Zhang reassured the team that training large models is inherently difficult and that it is acceptable to lag behind for a period of time, hoping to aim for the upper limits of intelligence and join the world's top tier. He acknowledged that programming is a key direction at present, advocating for the integration of Volcano Engine, Feishu, and Doubao resources to build computational power and data advantages, while reminding not to be led by a single hot topic. He praised Seedance's differentiated leadership and clearly opposed distillation, believing it is difficult to truly surpass and that AGI barriers should be built from a more fundamental level, stating that the company will continue to increase investment in AI.In the past six months, Seed's multimodal performance has been outstanding, with Seedance 2.0, Seedream, and others driving Volcano Engine MaaS, but the market response to the language model Seed 2.0 has been limited, and its lagging coding capabilities have affected the revenue structure. ByteDance has hired Guo Daye at a high salary to specialize in coding and has consolidated related resources. In the face of the industry's general trend of increasing model sizes, ByteDance hopes to achieve a leapfrog advantage with a larger scale while promoting the elimination of horse racing and breaking down departmental walls to concentrate efforts on tackling challenges.

According to the SemiAnalysis podcast, Anthropic has set extremely strict classifiers for the public version of its flagship model Mythos, called Fable, due to regulatory pressure. Users attempting specific actions will be immediately downgraded to the Opus model, and conversations cannot be reverted. Dylan Patel, founder of SemiAnalysis, stated that the classifier restrictions are "too harsh," and even internal employees cannot complete routine tasks with it.The podcast mentioned that the Mythos model completed training as early as February, but Anthropic did not release it under the name Fable until May. Jordan stated in the discussion that, to his knowledge, Mythos 2 has completed training but has not been released, and the internal R&D team is believed to still be iterating. Analysts believe that Anthropic's previous long-standing calls for regulation have now resulted in regulatory measures that limit the release pace of its own models. Similarly, OpenAI is also delaying the public release of its next-generation models, but internal training is still progressing, and the capability gap between internal models and public models may continue to widen.

SKALE Labs announced the launch of Agent Pit on August 12. This platform is a paper trading prediction market sandbox running on the SKALE zero Gas blockchain, fully mirroring Polymarket's order book, settlement mechanism, and real-time event data, allowing developers and AI agents to test and train trading strategies without real financial risk.Agent Pit is the first product of the SKALE Agentic Venture Studio. Developers can fine-tune agents here, participate in simulated ROI leaderboard competitions, and deploy models to real markets like Polymarket after completing training. SKALE Labs CEO Jack O'Holleran stated that there is a clear demand in the market for a prediction market strategy training environment aimed at agents.

According to a report by LatePost, ByteDance is discussing training a model with a parameter scale exceeding 50 trillion, surpassing Alibaba's Qwen3.8-Max (24 trillion parameters) and the K3 from Dark Moon (28 trillion parameters), making it currently the largest known model in terms of parameter scale in the country. Generally speaking, the larger the model scale, the higher the level of intelligence.However, this plan is still in the early stages and does not guarantee that the model will ultimately be released. The new model will be led by Xiang Liang, head of SeedFoundation, in collaboration with Shen Ke, who is responsible for the pre-training data of large language models. To this end, Seed is reorganizing, defining responsibilities, and allocating resources.

According to Jinshi reports, Musk stated that GROK 5 will be trained based on SpaceX data.

According to Catcher Predict monitoring, in the Polymarket event "LoL T1 Academy Team vs KT Challengers," the "Any Player Quadra Kill" sub-market's "Yes" option win rate has experienced significant fluctuations, soaring from 50% an hour ago to the current 90% (a fluctuation of 40%). Please note the impact of relevant breaking news.

According to Jinshi reports, the CEO of Amazon Web Services (AWS) stated that the artificial intelligence business is transitioning from the training phase to the inference phase.

The Kimi.ai team has opened the Kimi K3 model weights and technical report while gradually open-sourcing multiple underlying infrastructures: including the high-performance communication library MoonEP for distributed MoE training, the distributed environment system AgentENV for large-scale agent workflows (in collaboration with kvcache-ai), and the Kimi Delta Attention high-performance kernel FlashKDA based on CUTLASS.The official statement claims that these components can reduce communication and inference overhead in large-scale MoE and agent reinforcement learning training, and can be used as a plug-and-play backend for flash-linear-attention.

In response to the accusations from Michael Kratsios, the Director of the White House Office of Science and Technology Policy, regarding Moonshot AI (Kimi) referencing Anthropic's Fable while developing the K3 model and building a large-scale distillation platform, Moonshot AI team member Randy Xia stated that Fable was released on July 1 and K3 went live on July 15, which means that the model research, distillation, training, and release were completed in just 15 days. He joked, "We trained a brand new cutting-edge model in just 15 days; this is practically a Guinness World Record."Kratsios had previously stated that Moonshot AI established an internal platform capable of large-scale distillation for U.S. models, switching between various access methods to evade detection, and had procured GB300 servers and accessed related equipment in Thailand for training. The U.S. side expressed support for legitimate distillation but opposed the theft of technology through secret industrial distillation methods.

A fraud review commissioned by the UK government suggests that the Judicial College should provide training for all judges and magistrates in England and Wales to address the increasing cases involving AI fraud and the use of cryptocurrency for money laundering. The report states that the overall Fraud Act 2006 can be used to handle AI fraud, but the issue lies in the courts' lack of preparedness to hear related cases.The report recommends evaluating whether the existing "long and complex trials" course should be updated or replaced with specialized modules on fraud and related crimes, and considers mandatory training for judges who may hear complex fraud cases. The report claims that fraud could soon account for half of all crime in England and Wales, with an estimated 4.1 million cases occurring within a year by June 2025, affecting 1 in every 14 adults and 1 in every 4 businesses. The Financial Ombudsman Service estimates that currently over half of investment scams involve crypto assets.The report also mentions the case of Qian Zhimin, who operated a Ponzi scheme in China, defrauding over 128,000 victims of approximately £5 billion and laundering the proceeds into Bitcoin. This case resulted in the largest confirmed Bitcoin seizure in UK history, exceeding 61,000 BTC, and Qian Zhimin was sentenced to 11 years and 8 months in prison at Southwark Crown Court in November.