Recent Summaries

This Latent Space newsletter announces the discontinuation of SWE-Bench Verified as a reliable benchmark for coding AI models due to saturation and contamination. It features a discussion with Mia Glaese and Olivia Watkins from OpenAI's Frontier Evals team, who explain the reasons behind this decision and endorse SWE-Bench Pro as a more suitable alternative. The discussion also explores the future of coding evaluations, emphasizing the need for more complex, real-world tasks and human-intensive evaluation methods.

• Benchmark Contamination: Frontier models have been exposed to SWE-Bench problems during training, leading to models regurgitating solutions verbatim.
• Flawed Tests: Over 60% of the remaining problems in SWE-Bench Verified are deemed unsolvable due to overly narrow or overly broad test specifications.
• Endorsement of SWE-Bench Pro: OpenAI is officially moving away from SWE-Bench Verified and recommending SWE-Bench Pro as a more challenging and less contaminated benchmark.
• Future of Coding Evals: The focus is shifting toward longer-term tasks, open-ended design decisions, code quality, real-world product building, and human-intensive evaluations.
• Preparedness Framework: OpenAI's work on coding evals is tied to their Preparedness Framework, which aims to track and mitigate potential risks associated with advanced AI capabilities.

This article argues that the future of enterprise AI relies on interoperability between AI agents, moving away from siloed deployments to a collaborative ecosystem. It introduces Agent2Agent (A2A) as an open standard to facilitate cross-vendor communication and highlights the need for robust governance and trust to ensure responsible scaling.

• Interoperability is Key: The central theme is that AI agents must be able to communicate and coordinate actions across systems to unlock their full potential and avoid fragmented gains.

• Open Standards (A2A): The article promotes open protocols like A2A as essential for enabling seamless collaboration between agents from different vendors and technologies.

• Governance & Trust: The piece emphasizes the importance of transparency, auditability, and governance frameworks to ensure responsible and sustainable interoperability at scale.

• From Pilots to Operating Systems: Interoperability enables the transition from isolated AI pilots to AI-powered operating models that transform entire enterprises.

• Siloed AI agents lead to duplicated work, miscommunication, and bottlenecks, hindering enterprise-wide transformation.

• Interoperability requires open protocols, unified data fabrics, and centralized orchestration layers.

• Eaton's implementation demonstrates how interoperable AI agents can improve resolution times, reduce tickets, and enhance employee experience.

• A2A supports enterprise-grade authentication and auditability for robust governance.

• Prioritizing interoperability today is crucial for enterprises aiming to lead in AI-powered collaboration.

1. High-Level Overview: The newsletter focuses on the potential of custom ASICs (Application-Specific Integrated Circuits) for AI models, highlighting Taalas' impressive Llama 3.1 8B inference speed using custom silicon and discussing the economic viability of ASICs per model. It also covers recent developments in frontier model evaluations, particularly Gemini 3.1 Pro, and raises questions about the validity and consistency of AI benchmarks.

2. Key Themes/Trends:

   • Custom ASICs for AI: Exploring the idea of "baking" LLMs into silicon for faster and cheaper inference.
   • Frontier Model Evaluations: Examining the performance of Gemini 3.1 Pro and other models on various benchmarks.
   • Benchmark Reliability: Questioning the consistency and relevance of current AI benchmarks like SWE-bench and ARC-AGI.
   • Token Efficiency and Cost: Highlighting the importance of token efficiency and cost-effectiveness in frontier models.

3. Notable Insights/Takeaways:

   • Taalas' 16,960 tokens per second inference speed with Llama 3.1 8B using custom silicon demonstrates the potential of ASICs.
   • The economic argument for custom ASICs is strengthening, particularly for models with billion-dollar training runs.
   • While Gemini 3.1 Pro shows strong retrieval capabilities and token efficiency, it faces tooling and consistency issues.
   • SWE-bench Verified evaluation methodologies need standardization to ensure fair comparisons across labs.
   • Current benchmarks may not fully capture real-world performance, prompting a debate on what metrics truly matter.

This MIT Technology Review newsletter promotes an exclusive subscriber-only eBook titled "The Great AI Hype Correction of 2025," which reflects on the overblown promises of AI companies and the need to readjust expectations. The eBook is part of a larger "Hype Correction" series. It features articles and analysis that suggest a critical look at the current state and future of AI.

• AI Hype Correction: The overarching theme is a necessary correction of the excessive hype surrounding AI, particularly after a year of reckoning in 2025.

• LLM Limitations: The eBook challenges the notion that Large Language Models (LLMs) are a panacea and highlights the limitations of AI as a quick fix for all problems.

• Bubble Concerns: It raises questions about a potential AI bubble and explores its possible nature.

• Beyond ChatGPT: The content positions ChatGPT as just one point in AI's evolution, not the ultimate end point.

• The eBook argues that the AI industry needs to move beyond unrealistic promises and address the fundamental limitations of current AI technologies.

• It suggests a potential market correction in the AI sector.

• The analysis encourages a more grounded and realistic perspective on the capabilities and impact of AI, moving past the initial excitement surrounding tools like ChatGPT.

• The featured articles imply a growing backlash against AI, potentially fueled by concerns about its applications and connections to controversial figures.

This newsletter focuses on the release of Google's Gemini 3.1 Pro, positioning it as a competitive advance that surpasses previous models in certain areas. It summarizes the key aspects of the release, including performance benchmarks, practical applications, and the general sentiment surrounding its launch.

• Frontier Model Race: The newsletter highlights the continuous cycle of incremental updates among leading AI models, with Gemini 3.1 Pro being Google's latest offering to stay competitive.

• Benchmark Performance: Gemini 3.1 Pro demonstrates strong performance on benchmarks like ARC-AGI-2 (77.1%) and SWE-Bench Verified (80.6%), indicating improved reasoning, coding, and agentic capabilities.

• Practical Applications: The newsletter showcases Gemini 3.1 Pro's capabilities in SVG design and translating textual descriptions into visual aesthetics, demonstrating real-world improvements.

• Market Reaction: The release has generated mixed reactions, including excitement about practical improvements, skepticism about benchmark-targeting, and concerns about real-world agentic task performance.

• The release of Gemini 3.1 Pro appears to be driven by a need for Google to catch up with and potentially surpass competing AI models.

• While benchmark scores are impressive, the newsletter raises concerns about whether these translate into equivalent gains in real-world agentic tasks.

• The initial rollout has faced inconsistencies and availability issues, potentially impacting user experience.

1. Spatial Intelligence Startup World Labs Secures $1 Billion Funding: World Labs, founded by Fei-Fei Li, has raised $1 billion to advance its spatial intelligence technology, which focuses on generating editable and downloadable 3D virtual worlds from text or image prompts. The company's valuation has significantly increased since its initial funding in 2024, with major investments coming from Nvidia, AMD, and Autodesk.

2. Key Themes/Trends:

   • Spatial Intelligence as the Next Frontier: The focus on spatial intelligence signals a move beyond traditional AI that primarily processes language, towards AI capable of understanding and generating 3D environments.
   • Investment in Foundational AI: Major players like Nvidia, AMD, and Autodesk are investing in foundational AI companies like World Labs, reflecting a broader industry trend of securing access to cutting-edge technology.
   • Convergence of AI and Design Software: Autodesk's investment and advisory role indicate a growing convergence between AI and 3D design software, with potential applications in architecture, engineering, and manufacturing.
   • Generative AI for 3D Environments: The article highlights the increasing use of generative AI to create 3D virtual worlds, enabling rapid prototyping and development across various industries.

3. Notable Insights/Takeaways:

   • Fei-Fei Li's Leadership: Fei-Fei Li's involvement as founder adds significant credibility and expertise to World Labs, attracting substantial investment and attention.
   • Practical Applications of Spatial Intelligence: The potential applications of World Labs' technology span gaming, immersive media, robotics, simulation, architecture, and design, showcasing the broad applicability of spatial intelligence.
   • Autodesk's Strategic Investment: Autodesk's investment signifies the importance of "physical-world AI" to its future strategy and its vision of applying digital intelligence to design and construction.
   • Focus on Understanding Worlds, Not Just Words: Fei-Fei Li emphasizes the critical need for AI to understand geometry, physics, and dynamics, highlighting the shift towards AI systems that can reason about the physical world.