TL;DR

  • Document understanding is the unglamorous killer application - invoices, contracts, and scanned PDFs that were painful to extract data from are now tractable without dedicated pipelines
  • Vision models still under-deliver on precise spatial reasoning, object counting, and subtle medical or scientific imagery - these remain jobs for specialist models
  • Audio is the modality with the most upside: beyond transcription, it carries tone, pace, and hesitation that text loses, enabling fault detection, emotional analysis, and richer inputs
  • The teams getting real value treat multimodal as an invisible enabling capability within a workflow, not a feature to demo - and they verify high-stakes outputs just as they would text
  • The right question when evaluating multimodal is not “can we use this” but “what specific user problem becomes tractable that previously was not”

When the first multimodal frontier models shipped, the demos were genuinely impressive. A photo of a fridge interior with the model suggesting a recipe. A handwritten napkin sketch becoming working code. A short audio clip of a meeting being transcribed, summarised, and structured. It looked, briefly, like the boundary between modalities had collapsed and we were entering a new regime in which models could reason fluidly across text, images, and sound.

Two years later, the picture is more interesting. Multimodality has delivered real value, but not always where the demos suggested. It has also produced a long tail of features that are technically impressive and practically irrelevant, and a small set of capabilities that have quietly become indispensable to working systems. This post is the working summary of where the line falls.

What multimodal actually means in 2026

The state of the art in 2026 is foundation models that natively process images, text, and increasingly audio in a single pass, without separate translation steps between modalities. GPT-4o, Gemini 2, and Claude’s vision-capable models all operate in this regime, with audio support reaching parity for the major providers.

The important distinction from older approaches is that the model is not transcribing the image to text and then reasoning over the text. It is reasoning over the image directly, using a unified representation that lets it understand layout, relative position, visual style, and the kinds of things that are easy to see and hard to describe.

This unified representation is what makes the new capabilities possible, and it is also why some old assumptions about how to structure systems no longer hold. If you are still running OCR before sending text to a language model, you are using yesterday’s architecture for today’s problem.

Where vision actually earns its keep

After two years of production deployment, the use cases where vision-capable models are clearly worth the cost cluster into a few categories.

Document understanding. This is the killer application that has not been heavily marketed because it is unglamorous, but it is the one delivering the most value in enterprises. Invoices, contracts, forms, scanned PDFs - documents that have always been a pain to extract structured data from are now tractable. The model reads the document the way a human reads it, understanding the layout, the tables, the handwritten annotations, and produces structured output. The cost compared to dedicated document-AI pipelines is dramatically lower, and the accuracy on messy real-world documents is noticeably better.

UI and screen understanding. Models that can look at a screenshot and reason about what is on the screen have unlocked a new category of automation. Browser agents that can navigate websites without depending on brittle DOM selectors. QA tools that can verify visual regressions. Accessibility tools that can describe interfaces meaningfully. The reliability is not yet at the level where you can hand off mission-critical workflows, but for internal automation and developer tools the value is real.

Code from sketches. A whiteboard photo or a hand-drawn mockup turning into working code is one of the use cases that demoed beautifully and has held up. It is not a replacement for a designer. It is a way to compress the early-iteration loop between an idea and something runnable, and for early-stage product work it has become a normal part of the workflow. The latest generation of generation tools - covered in ChatGPT Images 2.0 - has also reshaped the design-side of the same loop.

Visual data extraction. Charts in PDFs, dashboards in screenshots, diagrams in research papers. The pattern of “I have a visual artifact and I need its underlying data as structured output” is now reliable enough to use in production pipelines. This was a substantial pain point in data engineering and it has quietly become tractable.

Image-grounded instruction. Asking the model questions about a specific image - “what does this error message say”, “is the cable plugged in”, “is this part defective” - is the most general capability and the foundation for many of the others. It is reliable enough for many production uses, with the standard caveat that you need to verify outputs for high-stakes decisions.

Where vision is still demoware

The use cases where vision models still under-deliver share a common shape: they require precise spatial reasoning, exact counting, or judgment under ambiguity that the model cannot execute reliably.

Counting objects in cluttered scenes. Models are surprisingly bad at “how many of X are in this image” once X gets above a small number or the scene is complex. They will produce confident wrong answers. Specialised computer vision is still the right tool here.

Precise measurement. Asking a model the dimensions of an object in an image, the angle between lines, or the exact position of a feature is unreliable. The model is doing approximate spatial reasoning, not measurement. If you need numbers you can trust, use a measurement system.

Subtle medical or scientific imagery. Despite the demos, current vision models are not reliable diagnostic instruments. They miss subtle pathologies, they confidently misread imaging, and they are not yet at the level where they should be making clinical calls without human expert review. Specialised medical imaging models trained for the task are dramatically more reliable.

Real-time video understanding at scale. Processing video frame-by-frame through a foundation model is expensive and slow. For genuine real-time video applications, specialised models are still the right architecture.

The common thread is that frontier multimodal models are generalists. They are excellent at the broad shape of a problem and unreliable at the long tail of edge cases that specialists handle well. For high-stakes vision tasks, the question is still whether the use case is in the well-served centre of capability or in the long tail where you need a specialist.

Audio is where the surprises live

Audio is the modality that has surprised me most in 2026. The capability has improved faster than I expected, and the value has shown up in places I did not anticipate.

The headline cases are well-known. Real-time transcription that is accurate enough for meeting notes. Conversational interfaces that feel natural rather than robotic. Voice agents that can hold a multi-turn conversation without the strange pauses that gave away the older systems. The provider landscape is broader than it was: I covered Grok’s voice APIs and the older OpenAI Voice Engine separately, and the parity across providers is now genuinely close.

Less well-known but more interesting is the use of audio as input for tasks that have nothing to do with speech. The model listening to an industrial machine and identifying a developing fault. The model analysing a recording of a customer support call to identify emotional escalation in real time. The model parsing the audio of a podcast to extract structured highlights without relying on a transcript intermediate.

The unifying property is that audio carries information that text loses. Tone, pace, hesitation, background context. A transcript flattens that into words. A multimodal model that processes the audio directly preserves it, and downstream reasoning has more to work with.

This is also where the responsibility considerations are sharpest. The same capabilities that let a model recognise emotional escalation can be used to surveil employees in ways that are deeply uncomfortable. The same capabilities that let an AI clone someone’s voice for accessibility purposes can be used for fraud and impersonation. Multimodal audio is genuinely powerful, and that power requires more deliberate thinking about deployment than text-only systems demanded.

The patterns that work in production

The teams I see getting real value out of multimodal AI in 2026 share a few patterns.

They use multimodal as a capability, not a product. The user-facing product is not “the AI can see images”. The user-facing product is the workflow it enables, with vision as an invisible enabling capability. Document upload that just works. Screenshots that turn into bug reports. Photos that become inventory entries. The vision is implementation detail.

They verify high-stakes outputs. Multimodal output goes through the same verification disciplines as text output. If the model extracts numbers from a chart, those numbers are validated. If it identifies an object in an image, that identification is checked or used in a way that bounds the cost of being wrong. The verification patterns I described in AI agents that actually work apply unchanged.

They cache aggressively. Vision tokens are expensive. The same image processed multiple times costs the same multiple times unless you are caching. Treating the visual representation of stable images as cacheable is a significant cost saver for any workflow that touches the same documents repeatedly. The general patterns I cover in Prompt caching: the quiet performance win apply equally to multimodal workloads, often with bigger absolute savings.

They reach for specialist models when the general one is not enough. A frontier multimodal model is the default. A specialist model is the right answer when the use case demands accuracy the generalist cannot deliver. The skill is knowing when to make the switch, and the answer is “the moment your error rate starts costing more than the engineering effort to integrate a specialist”.

They are honest about latency. Vision and audio both add latency to a request. For interactive applications, this matters. For batch document processing, less so. Latency-sensitive use cases either need careful engineering or need to be split into a fast path that uses cheaper models and a slow path that uses the multimodal one.

What I would tell someone evaluating multimodal today

If you are building a new product in 2026 and considering whether to use multimodal capabilities, the question is not “can we use this”. The question is “what specific user problem becomes tractable that previously was not”. If you have a clear answer, multimodal is probably worth integrating. If the answer is hand-wavy, you are probably building a feature for the demo rather than for the user.

The places to look first are document understanding, screen understanding, and audio analysis for tasks where the information in the modality is genuinely useful. These are where the value is most clearly real.

The places to be sceptical of are anything requiring precise spatial reasoning, anything requiring counting, anything safety-critical without a specialist model in the loop, and anything where the value is “AI now sees images” rather than a specific user benefit.

The state of the art has improved enough that the question is no longer “is multimodal capable enough”. It is “is the use case in the part of the capability surface that has matured”. Two years ago, that surface was small. In 2026 it is broad enough that most teams can find genuinely valuable applications, and the next two years will continue to expand it.

The mistake to avoid is treating multimodal as a thing to deploy because it exists. The mistake to embrace is treating it as a tool that, used in the right places, makes some previously-painful problems quietly tractable. The systems that age well will be built on that latter framing.