Why Is Understanding Large Language Models So Difficult?
By: Ben Brubaker
Source: QuantaMagazine
Usually, a machine is not built without understanding how it works; but for researchers working on large language models, it is precisely this understanding that has not yet been achieved. In fact, their work sometimes resembles gardening more than engineering.
Martin Wattenberg, a language-model researcher at Harvard University, says: «When you plant a tomato seed in the soil, a tomato plant grows. You water it and pull the weeds around it, but how does the plant really grow?».
These scientists try to analyze the behavior of language models by observing their reactions to different inputs – an approach similar to behavioral psychology. But researchers in the emerging field of «Mechanistic Interpretability», which is inspired by neuroscience, open up the model from the inside rather than observing it from the outside, in order to understand exactly what is happening within it. Their early efforts have managed to explain how language models represent concepts and how they perform some simple tasks. They have also produced unexpected findings that show how difficult a true understanding of AI is.
An artificial brain at the heart of large language models
Large language models are built on mathematical objects that are partly inspired by the human brain. These objects, known as artificial neural networks, run a chain of simple mathematical operations to process strings of numbers – which represent words. How a language model performs – whether its output is nonsense or highly accurate – depends on another set of numbers called parameters. These parameters express the relationships within the neural network.
Large language models can have billions or even trillions of parameters, and no one knows exactly how these values should be chosen; therefore, the model starts with random values, and a huge volume of data is fed into it with one simple goal: predicting the next word in each piece of text.
The model does this trillions of times. After each attempt, another algorithm adjusts the parameters so that the probability of producing the correct answer increases slightly. This process is called «training», but the word is a bit misleading; because once the process begins, the researchers’ role is roughly that of a gardener in a plant’s growth.
A window into the brain of language models
In theory, researchers can see all the parameter values of a trained model. They can also record how each internal part of the model reacts to a specific input – which is called activation. This data is equivalent to a complete map of a human brain along with electrodes to measure the activity of each neuron. But all this data does not necessarily lead to an interpretation. Predicting the model’s behavior toward new inputs remains challenging.
Fortunately, interpretability researchers do not settle for merely observing the values – they can also manipulate these values. Editing parameters is like performing brain surgery with extremely high precision. These researchers can also temporarily change the activations to see how the model’s output changes.
They can even copy a mental state and paste it into another scenario: they give the model the first input, store the activation of certain parts, and then insert those activations into the model’s response to a second input. This method helps researchers identify where certain concepts are stored in the model. But the results are not always straightforward. Even if a concept is stored in a particular part of the model, manipulating another part may change that concept – something that shows the inner workings of neural networks do not align with human intuition.
Asma Ghandeharioun, a researcher at Google DeepMind, says:
«There are many things that on the surface should be right, but when you look more closely, they are not at all.»
Steps toward better understanding
Researchers have managed to identify some of the processes that language models use to perform tasks such as retrieving previous words, recognizing grammatical roles, or doing simple calculations. However, models sometimes take different paths for similar tasks that seem completely random. It is like checking the weather before brushing your teeth, because if it rains you will use your bright-pink toothbrush!
In other cases, models have independent groups of similar components that do exactly the same job, and this makes efforts to separate the functions difficult. A phenomenon called «Emergent Self-Repair» has even been observed, in which disabling part of the model causes other components to change their behavior and take over the task of the disabled part.
A promising future within the complexity Despite all this complexity, many interpretability researchers remain optimistic about the future of this field. Wattenberg says:
«There is room for progress. We are now much further ahead than five years ago.»
