LLMs: what are large language models

When you ask an AI assistant to summarize a report or draft an email, the text is produced by an LLM. The acronym appears in almost every news story about artificial intelligence, but it is rarely explained.

This article shows what an LLM is, how it generates answers, how it is trained and which limits you should keep in mind when using it. For a broader overview first, see what is artificial intelligence.

What LLM stands for

LLM stands for large language model.

A language model is a program that has learned, from examples, which sequences of text are likely. According to the Hugging Face LLM course, models such as GPT, BERT and T5 are trained as language models on large amounts of raw text in a self-supervised way. The result is a kind of statistical portrait of the language.

The “large” in the name refers to two things: the amount of data used in training and the number of parameters in the model.

Parameters are the internal numbers the model adjusts during training. They work like millions or billions of small tuning knobs. Two documented examples help give a sense of scale:

• GPT-3, described in a 2020 paper by Tom B. Brown and co-authors, had 175 billion parameters;
• the largest model in the Llama 3 family, described in a 2024 paper, has 405 billion parameters and a context window of up to 128,000 tokens.

There is no official line separating a “large” model from a “small” one. In the editorial team’s assessment, the term LLM describes a family of techniques more than an exact size.

The core idea: predicting the next piece of text

An LLM does not look up a bank of ready-made answers. It generates text bit by bit.

First, the text you type is split into tokens, small pieces that can be whole words, parts of words or symbols. We explain this in detail in how tokens work in AI models.

Then the model calculates which tokens are most likely to come next. It picks one, adds it to the text and repeats the process. The whole answer is born from this cycle, token by token.

Imagine the sentence “It rained all day, so I took my.” A well-trained model assigns a high probability to “umbrella” and a low probability to “pineapple.” Not because it “understands” rain the way you do, but because it has seen similar patterns millions of times in the data.

Hugging Face calls this task causal language modeling: the output depends on what came before, never on what comes after. It is a simplified description, but enough to understand why an LLM sounds fluent: it was optimized precisely to produce plausible continuations.

Plausible, however, is not the same as true. We will come back to this point later.

The Transformer, the architecture behind it

Most current LLMs are built on the Transformer, a neural network architecture introduced in 2017 in the paper “Attention Is All You Need” by Ashish Vaswani and co-authors.

The novelty was to base the model entirely on attention mechanisms, abandoning the recurrent and convolutional structures that dominated the field. Put simply, attention lets each token “look” at the other tokens in the sequence and weigh which ones matter most for interpreting it.

In the sentence “The bank was crowded because it was payday,” attention helps the model link “bank” to “payday” rather than to the bank of a river.

According to the authors, the Transformer was more parallelizable and required much less training time than the alternatives of the day. That efficiency made it feasible to train ever larger models.

The original Transformer was designed for translation and had two parts: an encoder, which reads the input, and a decoder, which generates the output. Hugging Face explains that decoder-only models are the most widely used for text generation. This is the family of conversational assistants. For the bigger picture, see how generative AI works.

How an LLM is trained

Training usually has two major stages.

Pre-training

In pre-training, the model starts with random parameters and is exposed to an enormous volume of text. The task is always the same: predict the next token. Since the right answer is already in the text itself, nobody needs to label the examples by hand. That is why the process is called self-supervised.

To give an idea of the volume, the Llama 3 paper states that the model was pre-trained on about 15 trillion multilingual tokens.

Why so much? A 2020 study by Jared Kaplan and co-authors, “Scaling Laws for Neural Language Models”, observed that the error of language models falls predictably, following a power law, as model size, amount of data and the compute used in training increase. Results of this kind helped justify the race toward larger models.

The cost grows too. Hugging Face notes that training large models consumes a great deal of time, compute and energy, with environmental impact.

Tuning and alignment

A model that has only been pre-trained knows how to continue text, but does not necessarily follow instructions. Asking it to “summarize this contract” may produce some random continuation instead of a summary.

The InstructGPT paper, published in 2022 by Long Ouyang and co-authors, showed a way to solve this. The authors state that making a model bigger is not enough for it to follow the user’s intent. The proposal was:

1. tune the model with examples of answers written by people (so-called fine-tuning);
2. ask human raters to rank the model’s answers from best to worst;
3. use those preferences to train the model with reinforcement learning, a technique known as RLHF (reinforcement learning from human feedback).

According to the paper, raters preferred InstructGPT’s answers to GPT-3’s, even though GPT-3 was much larger. The authors themselves note that the model still made simple mistakes.

What LLMs do well

The GPT-3 paper drew attention to an ability that seems commonplace today: performing new tasks with only instructions and a few examples in the text of the request itself, without new training. The authors called this few-shot learning and cited translation, question answering and simple arithmetic among the tasks tested.

In practice, this explains why the same LLM can:

• summarize, rewrite and translate text;
• classify messages or extract data from documents;
• suggest and explain code;
• answer questions in natural language.

How you phrase your request makes a difference to the result. The subject is covered in prompt engineering.

LLMs are also the foundation of more complex systems, such as those that consult your own documents before answering (RAG) and those that carry out multi-step tasks (AI agents).

Limits you need to know

An LLM’s fluency is precisely what makes its errors hard to spot. These are the main risks.

False answers that look true. Models sometimes produce plausible but wrong statements, a phenomenon called hallucination. A 2025 paper by Adam Tauman Kalai and co-authors, “Why Language Models Hallucinate”, argues that part of the problem comes from how models are trained and evaluated: the most common scoring methods reward “guessing” over admitting uncertainty. See more in AI hallucinations.

Biases inherited from the data. Pre-training uses large volumes of text collected from the internet, and the model learns the good and the bad in that material. Hugging Face shows an example with the BERT model: when completing sentences about professions, it suggests different occupations for men and women, reproducing stereotypes. According to the course, fine-tuning with your own data does not eliminate this underlying bias.

Knowledge limited to training. What the model “knows” comes from the material it was trained on. Later facts only make it into the answer if they are supplied in the request itself or retrieved by a connected tool.

Finite working memory. Each model can only consider a limited amount of text at a time, the so-called context window. Learn how this affects long conversations in context and memory in AI.

Text that is hard to tell from human writing. The GPT-3 authors already reported in 2020 that human raters had difficulty distinguishing news articles generated by the model from those written by people. This makes it easier to use LLMs to produce disinformation at scale.

How to use an LLM more safely

Some practices reduce risks in everyday use. These are the editorial team’s recommendations, not technical rules:

• treat the answer as a draft, not as a verified fact;
• check numbers, dates, names and quotes against independent sources;
• be suspicious of references you cannot open: they may have been made up;
• avoid pasting personal data, passwords or confidential information without knowing how the service handles that data;
• for decisions about health, money or law, consult a professional.

For a more complete guide, read how to evaluate AI responses.

Key takeaways

An LLM is a statistical model trained on enormous amounts of text to predict the next token. From this simple task, repeated at large scale on the Transformer architecture, comes the ability to summarize, translate and hold conversations.

Tuning stages with human feedback make the model more useful at following instructions, but they do not make it infallible.

Use LLMs as first-draft assistants. The responsibility for checking what matters is still yours.