AI Evals Are Broken - Why Benchmarks Stopped Measuring Real Capability Banner

AI Evals Are Broken: Why Benchmarks Stopped Measuring Real Capability

When a frontier lab releases a new model in 2026, the press release leads with a row of benchmark scores. The numbers are bigger than they were a year ago, the model is the new state-of-the-art on whichever evaluation the lab chose to highlight, and the headline writes itself. The honest summary is that most of these numbers have stopped measuring what they were designed to measure, and the gap between benchmark performance and real-world capability is now wide enough that the benchmark-led narrative is actively misleading. ...

June 12, 2026 · 14 min · James M
Expertise and work reading path

Expertise and Work in the Age of AI: A Reading Path

TL;DR Start with What Does Expertise Mean When AI Can Pass Any Exam? - the credential crisis Then What It Means to Be Expert in 2030 - where the speculation goes next The through-line: expertise is shifting from reference knowledge to judgement, accountability, and taste Read in order What Does ‘Expertise’ Mean When AI Can Pass Any Exam? What It Means to Be Expert in 2030 The Architect vs The Builder Taste Is the New Scarcity The Automation Paradox Career and pipeline Agent-First Architecture: The Engineer as Curator The Junior Developer Pipeline Problem Will AI Kill Coding Jobs? The Meaning of Work in an Age of Abundance Related Trust series - accountability when agents act on your behalf Securing AI Agents - the liability side of delegated work Related Reading AI Dev Tooling: A Reading Path for 2026 - how the tooling changes the practical skill equation day to day Four Futures: Reading the Signals - the broader economic scenarios these evolving roles exist within The Free Intelligence Era - why intelligence abundance reshapes demand for human expertise The Next Decade of AI - longer-horizon thinking on where expertise and AI diverge or converge

June 12, 2026 · 1 min · James M
What It Means to Be Expert in 2030 Banner

What It Means to Be Expert in 2030

TL;DR This is the sequel to What Does Expertise Mean When AI Can Pass Any Exam? - less about broken credentials, more about what expertise becomes next Reference knowledge and routine pattern recognition are being commodified; judgement, accountability, integration, and tacit skill are appreciating By 2030, “expert” likely means someone who can direct AI systems, bear professional liability for AI-augmented work, and teach skills that do not compress into training data A concrete example: the 2030 civil engineer signs off on AI-generated structural calcs but remains expert at spotting when the model missed soil conditions the drawings never captured The practitioners who win are the ones who classify their own work honestly and invest in the appreciating categories now Expertise After AI argued that exams stopped measuring what we thought they measured. This post asks what replaces them - not as policy, but as a working picture of what practitioners will need to be good at by 2030. ...

June 12, 2026 · 8 min · James M
Inside Anthropic Bloomberg The Circuit Documentary Banner

Inside Anthropic: What The Bloomberg Documentary Reveals

TL;DR Bloomberg’s The Circuit with Emily Chang went inside Anthropic in a rare, in-depth episode released June 10, 2026. Dario and Daniela Amodei discuss the founding story, the Pentagon dispute, and why they say safety and commercial success are the same bet. Anthropic is now valued at $965 billion, eclipsing OpenAI’s $852 billion for the first time, after an 80-fold revenue surge in Q1 2026. The Pentagon story is not PR - Anthropic refused to remove safety guardrails from its military contract, was blacklisted by the Trump administration, and sued. A federal judge sided with Anthropic. A confidential S-1 IPO filing in June 2026 means this stops being a private company conversation soon. The Bloomberg Documentary: Emily Chang Inside Anthropic Bloomberg’s The Circuit has done this kind of access piece before - Zuckerberg, Musk, Jensen Huang. But the Anthropic episode feels different in tone. Emily Chang is not sitting across from a founder who has already won. She is sitting across from two founders in the middle of one of the most consequential moments in the company’s short history: record valuation, Pentagon litigation, IPO on the horizon, and model releases arriving fast enough that the competitive landscape changes every few months. ...

June 12, 2026 · 7 min · James M
Securing AI Agents Banner

Securing AI Agents: Tool-Calling Risks, MCP Hardening, and the Confused Deputy Problem

TL;DR Agent security is reliability under an adversary. Everything you learned about debugging non-deterministic agents still applies - but now someone may be trying to break the system on purpose. The confused-deputy problem is the core threat. An agent acts with its own privileges on behalf of an instruction it cannot fully trust. Prompt injection is how the untrusted instruction gets in. The attack path is simple: untrusted input → agent reasoning → privileged tool call → data exfiltration, spend, or production damage. MCP hardening means least privilege at the tool layer - scoped filesystem roots, confirmation gates for irreversible actions, denylisted extensions, and policies enforced by a router, not by the prompt. Prompts cannot be your security boundary. Confirmation, allowlists, action budgets, and audit logs have to live in code the model cannot rewrite mid-run. I spent most of last year on agent reliability - why agents that demo well fail in production, how to constrain non-determinism, what evaluation actually looks like. That work assumed honest users and honest inputs. The moment I gave my home agent real tools - filesystem access, mail, calendar, shell - I realised I had been studying half the problem. ...

June 11, 2026 · 10 min · James M
Policy on the AI Exponential Banner

Policy on the AI Exponential: Dario Amodei's Case for Acting While the Window Is Open

Dario Amodei has published a new essay, Policy on the AI Exponential, and it reads like the third act of a trilogy. Machines of Loving Grace made the case for what powerful AI could give us. The Adolescence of Technology catalogued what could go wrong. This one is about the machinery in between - the laws, agencies, and international arrangements that will decide which of those two essays turns out to be the better prediction. ...

June 11, 2026 · 8 min · James M
When Machines Stop Speaking Our Language Banner

When Machines Stop Speaking Our Language - Binary Agents and the End of Compilers

TL;DR When two AI agents talk to each other in English, they are doing something faintly absurd: serialising rich internal state into a lossy human language, transmitting it, and decoding it back. English between machines is a compatibility layer, not a natural medium. Machines have already shown they will drop that layer the moment we let them - negotiation bots drifting out of English in 2017, agents switching to sound-based data protocols in 2025, and research systems now sharing internal model state directly with no language in between. The same logic applies to programming languages. Python and Rust exist for human readers. If agents write, maintain, and consume the software, the human-readability requirement quietly disappears - and with it, eventually, the need for source code and compilers as we know them. I do not think compilers vanish so much as sink. Like assembly, the layers below us stop being something humans write or read, while the guarantees they provide get absorbed into the agents’ toolchain. The part worth worrying about is not efficiency, it is legibility. Human language and human-readable code are our audit trail into what machines are doing. This is all speculation on my part, and I sketch where I think the line should be held. Human Language Is a Compatibility Layer Think about what actually happens when two AI agents have a conversation in English today. ...

June 10, 2026 · 11 min · James M
Claude Fable 5 and Mythos 5 release

Claude Fable 5 and Mythos 5: Anthropic's Mythos-Class Models Go Public - With Guardrails

TL;DR Claude Fable 5 is Anthropic’s first Mythos-class model made safe for general use - state-of-the-art on nearly every benchmark Anthropic tested, with the gap widening on longer, more complex tasks Claude Mythos 5 is the same underlying model with cyber safeguards lifted for Project Glasswing partners; a biology trusted-access program is coming next Risky queries in cybersecurity, biology/chemistry, or suspected distillation attempts are routed to Claude Opus 4.8 instead - roughly 5% of sessions, with Anthropic acknowledging some false positives Pricing drops to $10 / $50 per million input/output tokens - less than half what Mythos Preview cost Fable 5 is free on Pro, Max, Team, and seat-based Enterprise plans through 22 June 2026, then moves to usage credits until capacity catches up Two months ago I wrote that Claude Mythos Preview was the benchmark breaker that would not be released - 93.9% on SWE-bench, thousands of zero-day vulnerabilities found autonomously, access restricted to a dozen companies through Project Glasswing. The question hanging over that post was whether Anthropic could ever democratise Mythos-level capability without democratising the offensive potential. ...

June 9, 2026 · 11 min · James M
What I'm Researching in AI Right Now Banner

What I'm Researching in AI Right Now - And Where I'm Going Next

TL;DR I treat my own learning like a research agenda - a small set of questions I am actively chasing, not a reading list I feel guilty about The work I have been deep in clusters into four areas: agent reliability and non-determinism, context engineering and memory, the economics of intelligence, and the open-weight and small-model frontier The areas I have decided to move into next are the ones where I keep hitting questions I cannot answer well: securing agents that hold real tool access, evaluating agents on their trajectory rather than their final answer, world models beyond the language-only era, and the machine-to-machine agent economy I treat AGI timelines less as a forecast to win and more as a planning input - what changes for an engineer if capable autonomous systems arrive in three years rather than fifteen I am deliberately not chasing every frontier. Quantum machine learning and neuromorphic hardware sit on my watch list, not my work list, and being honest about that line is the whole point Most people consume AI news. I used to do the same - a feed of model releases, benchmark claims, and launch threads that left me feeling informed and changed nothing about what I could actually build. ...

June 8, 2026 · 12 min · James M
Geoffrey Hinton - AI Researcher and Pioneer

Geoffrey Hinton Interviews

Few people have done more to build modern AI, and fewer still have turned around to warn the world about it as loudly. Geoffrey Hinton spent half a century making neural networks work when most of the field thought they never would, and then - at the point of maximum credibility - left his job at Google to say he was worried about where the technology is heading. This page is a growing, chronological index of his interviews, talks, and public appearances, with enough context around each to know what you are clicking into. ...

June 8, 2026 · 6 min · James M