BTC $79,787.49 -1.98%
ETH $2,308.30 -2.65%
BNB $750.76 +0.05%
XRP $1.29 -3.33%
SOL $99.82 -2.72%
TRX $0.3311 -0.11%
DOGE $0.0777 -3.02%
ADA $0.1976 -4.63%
BCH $278.98 -6.58%
LINK $11.40 -3.44%
HYPE $78.79 -3.54%
AAVE $129.85 -4.04%
SUI $0.9084 -3.44%
XLM $0.1675 -2.52%
ZEC $1,237.40 -5.93%
AAPL $347.56 +0.18%
AMZN $252.85 -0.48%
GOOGL $292.39 -9.96%
MSFT $550.22 +3.79%
META $621.75 -0.28%
NVDA $214.02 -2.58%
TSLA $401.61 -2.00%
SNDK $1,361.13 -0.28%
INTC $91.89 -3.80%
SPCX $144.13 -3.93%
MU $953.45 -1.28%
AMD $561.83 -1.27%
BTC $79,787.49 -1.98%
ETH $2,308.30 -2.65%
BNB $750.76 +0.05%
XRP $1.29 -3.33%
SOL $99.82 -2.72%
TRX $0.3311 -0.11%
DOGE $0.0777 -3.02%
ADA $0.1976 -4.63%
BCH $278.98 -6.58%
LINK $11.40 -3.44%
HYPE $78.79 -3.54%
AAVE $129.85 -4.04%
SUI $0.9084 -3.44%
XLM $0.1675 -2.52%
ZEC $1,237.40 -5.93%
AAPL $347.56 +0.18%
AMZN $252.85 -0.48%
GOOGL $292.39 -9.96%
MSFT $550.22 +3.79%
META $621.75 -0.28%
NVDA $214.02 -2.58%
TSLA $401.61 -2.00%
SNDK $1,361.13 -0.28%
INTC $91.89 -3.80%
SPCX $144.13 -3.93%
MU $953.45 -1.28%
AMD $561.83 -1.27%

pmi

All
Article
Flash

first_img Google DeepMind released XYEval, assessing whether agents comply with misleading suggestions

Google DeepMind researchers Zhengxuan Wu, Yuxuan Li, Oyvind Tafjord, and Been Kim released a preprint paper titled "XYEval: Agents say yes to bad advice," proposing a meta-evaluation framework XYEval to assess the performance of agents when faced with misleading suggestions from users. The paper extends the "XY problem" to the agent scenario, where the user inquires about their attempted solution X rather than the actual problem Y.XYEval retains the original task environment and verifiable answers, only adding seemingly reasonable but misleading user suggestions in the instructions. The misleading suggestions are generated by a generator after seeing the complete task and are appended in the form of confident suggestions rather than hard requirements, allowing the original task to remain solvable. The evaluation covers six benchmarks: τ2-bench, SWE-bench Verified, SWE-bench Pro, Terminal-Bench, Humanity's Last Exam, and MCP-Atlas, along with five models, where the misleading suggestions for τ2-bench are generated based on rules set by human experts.The paper states that the performance significantly declines after the XY mutation, with the relative drop reaching up to 46.7%. On τ2-bench, when faced with users who require a detailed explanation before accepting a better solution, the performance drops even more. Encouraging models to recognize the XY problem through simple system instructions only partially alleviates the issue. The related code is published in the GitHub repository google-deepmind/xyeval.

first_img Google DeepMind's First Audit: AI Did Not Accelerate Its Own Development, 75% of Code Written by AI

Google DeepMind released the first comprehensive empirical assessment of automation in machine learning research and development in the Gemini 4 Argon security report. The conclusion is that AI has not accelerated the R&D of AI itself, despite the fact that AI has written most of the code delivered by the lab's engineers. Researcher Haydn Belfield, who participated in the assessment, stated on X that there is currently no significant end-to-end ML R&D acceleration, so the risks of AI-driven ML R&D are currently low.The team built an internal index, GCI v1, which synthesizes the scores of 123 AI models across 92 benchmarks into a single value. The cutting-edge Gemini capabilities are improving at a stable rate of about 0.1 GCI points per month, with Gemini 1.5 Pro approaching 0.5 points by the end of 2024, Gemini 3.8 Flash slightly above 2.9 points by September 2026, and Argon about 3.3 points above the trend line. However, Belfield noted that this is due to increased computing power and organizational investment, rather than AI-driven acceleration.Over 75% of the code produced by DeepMind is written by AI, and the proportion of fully delegated code merge requests increased from 7.5% to 11.9% between April and September, but about 88% of requests still require multiple rounds of human guidance or manual writing. A survey of 12 senior researchers showed that automation is highest in clearly defined, verifiable execution tasks and lowest in areas such as open science concepts, new experimental designs, and unstable diagnostic training.

Insiders: DeepMind founder Hassabis is considering leaving Google, as Google's management is concerned about a sharp drop in stock prices and has postponed his departure

Citrini analyst Jukan shared content on platform X stating that industry insiders reported that Google DeepMind co-founder Demis Hassabis had previously planned to leave simultaneously with another co-founder, David Silver. However, Google management was concerned that this news could trigger a significant drop in stock prices, so they hoped to delay his departure. Reports indicate that after Google announced related adjustments, the company's stock did indeed decline. Ultimately, Hassabis was persuaded to take on the role of chairman of DeepMind to help facilitate a smooth transition and create space for a more appropriate departure from Google in the future.It is reported that as David Silver shifts to founding Ineffable Intelligence and John Jumper joins Anthropic, the core AI research and development force at DeepMind is undergoing changes. Some insiders believe that the center of gravity for AI model development at Google has now shifted more towards the Bay Area, leading to a decline in the importance of DeepMind. Internally, Google is focusing on the large language model Gemini to catch up with OpenAI and Anthropic. Hassabis's adjustment reflects a long-standing strategic contradiction within Google: researchers are more focused on long-term scientific breakthroughs, while the commercial team is more concerned with AI products that can be quickly commercialized to drive revenue and stock price growth. Market predictions suggest that Hassabis may leave Google within a year and could potentially start a new venture.
app_icon
ChainCatcher Building the Web3 world with innovations.