Multimodal LLMs Explained: How Models Understand Text, Images, Audio and Video, and Where They Still Fail

A clear guide to multimodal large language models: pipeline versus natively multimodal designs, the research milestones from CLIP to GPT-4o and Gemini, how current assistants handle images, audio and video, practical uses, known weaknesses, benchmarks and security risks.

PUBLISHED // UPDATED
DURATION // 06 MIN READ
Multimodal LLMs Explained: How Models Understand Text, Images, Audio and Video, and Where They Still Fail
FIG.01 // VISUAL DOCUMENTATION Multimodal LLMs Explained: How Models Understand Text, Images, Audio and Video, and Where They Still Fail

For most of the large language model era, AI assistants worked with text: you typed words and got words back. Images, speech and video were handled by separate systems joined together in pipelines. Multimodal models change that by accepting several kinds of input, and sometimes producing several kinds of output, within one model. Every major assistant now accepts images and documents, and most handle voice. This guide explains how multimodal models work, how they developed, what they are good at and where they still fail.

Pipeline versus natively multimodal

There are two broad ways to build a system that handles several modalities.

A pipeline chains specialised models: speech recognition turns audio into text, a language model reasons over the text, and a text-to-speech model reads out the answer. Pipelines are modular and easy to upgrade, but information is lost at each handoff. Tone of voice, background sounds or the layout of an image can disappear when everything is converted to text.

A natively multimodal model processes different input types within the same network, usually by converting images, audio and video into tokens or embeddings that share a representation with text. OpenAI’s GPT-4o system card, for example, describes an “omni” model trained end to end across text, vision and audio, so all inputs and outputs are processed by the same neural network. OpenAI reported that it could respond to audio in as little as 232 milliseconds, averaging 320 milliseconds, similar to human conversational response times.

Many products combine both approaches, so “multimodal” on a product page does not tell you which architecture is underneath.

Research milestones

YearMilestoneWhy it mattered
2021CLIP (OpenAI)Learned a shared embedding space for images and text from 400 million image-text pairs
2022Flamingo (DeepMind)Connected a vision encoder to a frozen language model for few-shot visual tasks
2023LLaVAShowed that visual instruction tuning could create capable open vision-language assistants
2023Gemini 1.0 (Google)Trained multimodal from the start rather than adding vision later
2024Gemini 1.5 (Google)Reasoned over millions of tokens of context, including hours of video and audio
2024GPT-4o (OpenAI)Single model across text, vision and audio, enabling real-time voice conversation

Sources: CLIP, Flamingo, LLaVA, Gemini and Gemini 1.5.

How today’s assistants compare

The major assistants have converged on a common baseline: image and document understanding, voice conversation and, increasingly, image generation. They differ in emphasis.

Assistant familyMultimodal emphasis
OpenAI (ChatGPT)Image understanding, voice conversation, integrated image generation
Google (Gemini)Long-context understanding of video and audio, tight links to YouTube, Photos and Android
Anthropic (Claude)Image and document analysis such as charts, screenshots and PDFs; text output
Apple (Apple Intelligence)On-device processing of screen content, photos and voice, with a larger model on Apple’s servers for harder requests

Capabilities change with each model release, so check current documentation before relying on any specific feature. Google’s Gemini 1.5 report showed one distinctive strength of long-context models: near-perfect retrieval of details across hours of video and audio within a single prompt, which enables questions such as finding the moment something happens in a long recording.

How multimodal models are trained

A common recipe is to start from strong components and connect them. A vision encoder (often CLIP-like) and an audio encoder convert inputs into embeddings, a projection layer maps them into the language model’s space, and the combined system is trained on paired data such as images with captions, documents with questions, and transcribed speech. Instruction tuning then teaches the model to follow requests about images or audio. Natively multimodal models go further by training on mixed-modality data throughout.

The main constraints are data and compute. High-quality paired data for video and audio is scarcer and costlier to process than text, and long videos produce very large numbers of tokens.

Where multimodal models are useful

Document understanding. Reading invoices, receipts, charts, scanned forms, slides and screenshots is one of the most practical gains. Tasks that once required brittle OCR pipelines and custom templates can often be handled by a general model, although accuracy should be checked on your own documents.

Accessibility. Apps such as Be My Eyes use vision-language models to describe scenes and answer questions for blind and low-vision users, such as reading a label or describing what is on a table.

Search across media. Video editors and archivists can search footage by description, and long-context models can summarise or find moments in recordings.

Voice interfaces. Real-time voice modes allow natural spoken conversation, including interruptions, which suits hands-free use, language practice and customer service.

Software and support. Screenshots of errors or interfaces give models context that is hard to describe in words.

Healthcare research. Combining images, records and notes is an active research area, but clinical use requires validation and regulatory approval; general assistants are not medical devices.

Known weaknesses

Multimodal models still make characteristic mistakes:

  • Counting and precise spatial reasoning, such as how many objects there are or which is to the left of another.
  • Small or dense text in images, especially at low resolution.
  • Long video, where models may sample frames and miss brief events.
  • Hallucinated details, describing things that are not in the image with confidence.
  • Audio in noisy settings, overlapping speakers and less common languages or accents.

Measuring capability

Benchmarks such as MMMU, which tests college-level reasoning across disciplines using charts, diagrams and images, give a rough comparison. Video and audio benchmarks are less mature, so vendor claims in those areas are harder to compare. The best test is a set of your own images, documents or recordings with known correct answers.

Security and privacy

Multimodal input widens the attack surface. Instructions can be hidden in images, documents or web pages that an assistant reads, a form of prompt injection. Agents that act on screenshots or browse the web are particularly exposed. Images and voice recordings can also contain sensitive information, from faces and locations to health details, so organisations should apply the same data-handling rules as for text, and check retention settings.

Our analysis

Multimodality has moved from a headline feature to a baseline expectation. The most valuable gains today are practical: reading documents and screenshots, assisting blind and low-vision users, searching media and enabling natural voice interaction. Weak spots remain in counting, spatial reasoning, small text and long video, and multimodal input creates new prompt-injection risks. Teams should test on their own material, keep humans in the loop for consequential decisions and treat images and audio as sensitive data.