Multimodal AI: text, images, video, voice and sensors

You photograph a medicine label and ask the assistant what it is for. Or you record a voice message asking for a summary of a video. In both cases, the system needs to understand something that is not typed text. This capability is called multimodal AI.

This article explains what a modality is, how multimodal models are built, what each type of data requires and which risks appear when AI starts to “see” and “hear”.

What a modality is

A modality is a type of data with its own structure. Text is a sequence of words. An image is a grid of pixels. Audio is a wave that varies over time. Video is a sequence of images, almost always accompanied by audio. Sensors produce continuous numbers, such as position, temperature, acceleration or distance.

A system is multimodal when it takes in or generates more than one of these modalities. A model that only reads and writes text is unimodal, however capable it is. If you are not yet familiar with the basics, start with how generative AI works and what LLMs are.

In practice, the word covers different situations:

• Multimodal input, text output. You send a photo, a scanned PDF or audio and receive a written answer.
• Text input, output in another modality. You describe a scene and receive an image or audio.
• Input and output in several modalities. The model listens and answers by speaking, or combines image and text in its answer.

The central idea: turning everything into something comparable

A language model works with tokens, small units of text converted into numerical vectors (see how tokens work). The trick of multimodal AI is to do the same with other types of data: cut them into pieces and convert them into vectors the model can process together with text.

A milestone for this idea was the Vision Transformer, from 2020. The paper showed that an image can be split into small squares (16 by 16 pixels, per its title) and each square can be treated as a “word” by a Transformer-family model, the same architecture used in language models (Dosovitskiy et al., 2020). According to the authors, this model, pre-trained on large datasets, achieved results competitive with the networks that had dominated computer vision until then.

Learning images from captions

Another important step was teaching the model to link images to natural-language descriptions. CLIP, from 2021, was trained on 400 million image-text pairs collected from the internet, on a simple task: figuring out which caption goes with which image (Radford et al., 2021).

The practical effect is that language becomes an index for visual concepts. The authors report that the model could classify images for tasks it had not been specifically trained on, so-called zero-shot learning, and they tested this on more than 30 computer vision datasets.

Attaching vision to a language model

The next generation connected these image encoders to already-trained language models. Flamingo, from 2022, connected pre-trained vision and language models and accepted sequences mixing images, videos and text (Alayrac et al., 2022). LLaVA, from 2023, also connected a visual encoder to an LLM and was fine-tuned on image-instruction examples that had been generated by a text model (Liu et al., 2023).

An academic survey of these systems, called multimodal large language models (MLLMs), sums up the architecture with an image: the LLM acts as the “brain” that performs multimodal tasks (Yin et al., 2023). The survey describes three typical modules. A pre-trained encoder compresses the image or audio into compact representations. A trainable connector translates that output into the LLM’s “internal language”. And the LLM handles understanding and reasoning. Some systems add a generator, which allows answering with images or audio, not just text.

Modality by modality

Images and documents

This is the most mature modality in general-purpose assistants. It is used to describe photos, read charts, interpret screenshots and extract text from scanned documents. CLIP already included recognizing text in images among the tasks tested.

The point to watch is that reading an image is not the same as understanding it. Small numbers, dense tables and handwriting remain a frequent source of error, and the model does not always say when it could not read something.

Voice and audio

There are two ways a system can handle voice.

The first is a pipeline: one model transcribes speech into text, an LLM responds in text and another model converts the answer into speech. Whisper, from 2022, is an example of a transcriber. It was trained on 680,000 hours of audio across several languages and tasks, with transcripts found on the internet (Radford et al., 2022).

The second is a single model trained end to end. The GPT-4o system card, published by OpenAI in October 2024, describes a model that takes text, audio, image and video as input and generates text, audio and images, with a single neural network processing everything (OpenAI, 2024). According to the company, the model can respond to audio input in as little as 232 milliseconds, with an average of 320, close to human response time in a conversation.

The difference matters. In a pipeline, everything that does not become text is lost along the way, such as tone of voice, pauses and background noise. In a single model, this information can, in principle, be taken into account.

Video

Video is expensive. One hour of images and sound produces a huge amount of data, and the model has to choose what to look at. Google’s Gemini API documentation shows how this is done in practice: by default, video is sampled at 1 frame per second, each frame consumes a fixed number of tokens and audio is processed separately, at 32 tokens per second (Google, video understanding documentation, consulted on October 8, 2026).

The documentation itself warns that, at this rate, fast actions may lose detail. If something happens between two sampled frames, the model simply did not see it. This applies to any system that samples video, and it is a good reason to be wary of answers about very brief events.

Sensors and the physical world

The modality least visible to the public is sensors: robot cameras, position measurements, machine readings. PaLM-E, from 2023, is an example of research in this direction. The model received “multimodal sentences” that interleaved images, continuous state estimates (such as the position of objects) and text, trained jointly with a pre-trained language model (Driess et al., 2023). The largest version had 562 billion parameters.

Here errors carry a different weight. A wrong answer about a photo can be corrected. A wrong action by a robot or an industrial system can cause physical harm. That is why this application usually goes hand in hand with the debate about AI agents and the limits of autonomy.

What it is used for day to day

In the editorial team’s assessment, the most common uses today fall into these groups:

• Documents and forms. Reading invoices, scanned contracts, tables in images and screenshots.
• Accessibility. Describing images for people with visual impairments and transcribing audio for people with hearing impairments.
• Customer service. Receiving a photo of a defective product or a voice message from a customer. We cover the risks of this use in AI in customer service.
• Search and organization. Finding images or video clips from a text description, exactly what CLIP’s link between images and captions makes possible.
• Inspection and industry. Analyzing images of parts, crops or facilities, generally with specialized models and human validation.

Where multimodal AI gets it wrong

Multimodal models inherit the problems of text models, such as AI hallucinations, and add new ones.

Seeing what is not there. A 2024 survey on the subject defines multimodal hallucination as generating responses inconsistent with the visual content (Bai et al., 2024). Typical examples: mentioning an object that does not appear in the image, getting the number of items wrong or inventing the relationship between them.

Missing what went by quickly. As shown above, videos are sampled. What happens between frames may never reach the model.

Obeying what is written in the image. The generative AI security project of OWASP, a reference organization in application security, lists prompt injection as the top risk for LLM applications. In the multimodal case, it warns that an attacker can hide instructions in images that accompany harmless text, and that cross-modal attacks can be hard to detect and mitigate with current techniques (OWASP, LLM01:2025). In practice, an image sent by third parties should be treated as untrusted data, not as a command.

Cost and context. Images, audio and video consume many tokens. This makes usage more expensive and takes up the model’s context window, a topic we cover in detail in context and memory in AI.

Privacy. Photos and recordings usually contain more information than it seems: faces, license plates, documents in the background, other people’s voices. Before sending, it is worth asking what else is in the image or audio.

How to use it more safely

1. Ask what the model saw before what it concluded. For example: “transcribe the numbers in the table” and only then “calculate the total”. That makes it easier to find the error.
2. Check critical details at the source. Amounts, dates, names and dosages read from an image should be verified against the original.
3. Send the best possible version. A sharp, well-lit photo cropped to what matters reduces the margin of error.
4. Be wary of answers about fast events in video. If the answer depends on a single moment, watch the clip.
5. Treat third-party images and audio as untrusted. In systems that process incoming files, instructions inside them should not trigger automatic actions.

Key takeaways

• Multimodal AI works with more than one type of data: text, images, audio, video or sensors.
• The common technique is to convert each type of data into representations a single model processes, almost always with an LLM at the core.
• Voice can be handled through a pipeline (transcribe, respond, synthesize) or by a single model; the latter preserves more of what is not text.
• Video is sampled, and what happens between frames can go unnoticed.
• Risks include describing what does not exist, obeying instructions hidden in images and exposing personal data. For decisions that matter, human checking is still necessary.