ChatGPT was launched by OpenAI on 30 November 2022 and quickly became one of the fastest-growing technology products in history. Millions of people use it to write, study, code and get answers. But how does ChatGPT work on the inside? This guide explains it in plain language.
What ChatGPT is
ChatGPT is a conversational assistant built on a large language model, or LLM. GPT stands for Generative Pre-trained Transformer: a generative, pre-trained model built on the Transformer architecture, introduced by Google researchers in 2017 in the paper Attention Is All You Need.
Step 1: text becomes tokens
The model does not read words the way we do. Text is split into small pieces called tokens, which can be a whole word, part of a word or a punctuation mark. Each token is converted into numbers the model can process.
Step 2: predicting the next word
At the centre of everything is a task that is simple to describe: given a text, predict which token comes next. The model chooses a likely token, adds it to the text and repeats the process, one token at a time, until the answer is complete. The Transformer architecture uses a mechanism called attention, which lets the model consider every part of the text at once to decide what is relevant.
Step 3: pre-training
To learn to predict well, the model is trained on an enormous amount of text. During training it adjusts billions of internal parameters to get the next word right more and more often. In the process it absorbs patterns of language, facts, styles and ways of reasoning found in the text.
Step 4: learning to be an assistant
A model that is only pre-trained completes text, but does not necessarily answer helpfully. So there is a second stage: the model is fine-tuned with example conversations and with ratings from people who indicate which answers are better. This technique is known as reinforcement learning from human feedback. It is what makes ChatGPT follow instructions, refuse dangerous requests and reply in a conversational format.
Why ChatGPT makes mistakes
Because the model generates the most likely text rather than looking facts up in a database, it can produce convincing but wrong answers, known as hallucinations. It also has a knowledge cut-off set by the end of training and only brings in current information when it uses tools such as web search. That is why data, figures, quotes and references should always be checked.
What ChatGPT does well
Summarising and reorganising text, suggesting ideas, explaining concepts in different ways, translating, editing writing and helping with programming are tasks where it is usually useful, especially when the person can judge the result.
How to use it better
Give context: say who you are, what the answer is for and what format you expect. Ask the model to show its reasoning or list its sources. Break large tasks into steps. And treat the answer as a draft to review, not as ready-made truth.
Privacy
Avoid pasting confidential information, other people's personal data or company secrets. Check in the settings whether your conversations may be used to train models and use the available controls. Companies should prefer business versions with data protection contracts.
To understand the technical foundation, read our guide to machine learning explained. To keep up with new developments, see our artificial intelligence news.