Multimodal

TL;DR Language-only models do not contain a reliable simulator of physical reality - they contain a statistical shadow of one, good enough for many tasks and dangerously wrong for others. A world model is a system that learns to predict how an environment evolves and can plan inside that prediction - not just describe it in text. The gap matters for agents that must act in physical space, manipulate objects, or reason about counterfactuals where the answer is not in the training corpus. The 2026 frontier includes generative world simulators, vision-language-action models for robotics, and sim-to-real pipelines - not one breakthrough but a stack assembling in parallel. For builders today: language agents with MCP tools are the right architecture for knowledge work. World models are the path to agents that can competently act in the physical world. Almost everything I have written about AI agents assumes a model whose understanding of the world arrives through text. That assumption has carried the field a long way. Context engineering, tool use via MCP, memory across sessions - all of it sits on top of language models that read, reason, and call APIs. ...

TL;DR Document understanding is the unglamorous killer application - invoices, contracts, and scanned PDFs that were painful to extract data from are now tractable without dedicated pipelines Vision models still under-deliver on precise spatial reasoning, object counting, and subtle medical or scientific imagery - these remain jobs for specialist models Audio is the modality with the most upside: beyond transcription, it carries tone, pace, and hesitation that text loses, enabling fault detection, emotional analysis, and richer inputs The teams getting real value treat multimodal as an invisible enabling capability within a workflow, not a feature to demo - and they verify high-stakes outputs just as they would text The right question when evaluating multimodal is not “can we use this” but “what specific user problem becomes tractable that previously was not” When the first multimodal frontier models shipped, the demos were genuinely impressive. A photo of a fridge interior with the model suggesting a recipe. A handwritten napkin sketch becoming working code. A short audio clip of a meeting being transcribed, summarised, and structured. It looked, briefly, like the boundary between modalities had collapsed and we were entering a new regime in which models could reason fluidly across text, images, and sound. ...