What We Know
In the traditional LLM training this happens:
You give some text (a book page etc.) to the learning circuit as the input. Say, 4096 words. The circuit itself is initialized with random rules (“weights”).
The circuit, for each word, makes a guess what the next word should be, based on all the previous words. For example, if word 51 (out of 4096), is “white”, the circuit guesses that #52 is “tree”. However, word 52 is actually “rabbit”. The guess is, as expected, wrong, because all the rules are random. And so on for all 4096 words.
Then we calculate the ‘difference’, the magnitude of error, between each guessed and the actual word, and express that difference as a number. That gives us 4096 values, usually 32-bit floating point numbers
These 4096 32-bit (4-byte) numbers are the only link between guesses and the reality, there is no other.
Then these numbers (‘loss’), total of 16 Kb, are fed back to the circuit, and all initially random rules are a bit adjusted, using these numbers. Then everything is repeated (with slightly adjusted rules), and new 4096 guesses are made, and new ‘loss’ numbers are calculated, etc.
The above is repeated hundreds, thousands, millions times.
After some time, the guesses get better. And then they get really good.
This should not happen.
It is amazing that 16 kilobytes, derived from context-wide dataset (resulting in context-wide number of embeddings), can be used to tune/adjust context_width x layer_depth number of large matrices. Billions of ’weights’.
Does anyone actually understand why this works?
The Official Story (what they teach) and Uncomfortable Mysteries
“The loss provides a scalar objective, and backpropagation computes how each parameter should change to reduce that loss. The chain rule of calculus makes this mathematically rigorous.”
The Uncomfortable Truth: Nobody actually understands why this works at scale.
Here’s what we don’t know (using 200B model as example):
MYSTERY #1: The Gradient Contains Enough Information
The Math:
Loss L = 4096 32-bit floating point numbers
Gradients: ∂L/∂W = [gradient for each of 200 billion parameters]
Update: W_new = W_old - learning_rate × ∂L/∂WThe Mystery:
One forward pass processes 4094 tokens
Produces 4096 numbers.
Somehow this contains enough information to improve 200B parameters
This shouldn’t work!
Information Theory Perspective:
Input: 16 kb of information (loss as float32)
Output: Meaningful adjustments for 200B parameters (400 Gb)
Where does the information come from?
MYSTERY #2: Why Don’t Gradients Conflict?
The same loss collection helps, simultaneously, to:
fix wrongly predicting ‘tree’ instead of ’rabbit’ at position 51
fix wrongly predicting ‘sat ’instead of correct ‘cat’ at position 1123,
and so on for all 4096 predictions.
At the same time.
Somehow the massive averaging doesn’t destroy information.
MYSTERY #3: Why Does Scale Help?
Observed Reality:
1B parameters: Mediocre
10B parameters: Good
100B parameters: Better
1T parameters: Amazing (presumably)
Standard Explanation: “More parameters = more capacity = better representation”
Problems with This:
Optimization should get harder: More parameters = higher dimensional space = harder to find good solution
More conflicts: More parameters getting contradictory gradients
Overfitting should dominate: With 200B parameters and “only” 1T tokens of training data, should memorize, not generalize
Reality: Bigger models generalize BETTER (which violates classical ML theory)
Suspected Truth:
Current theory is fundamentally wrong
Large models work for reasons we don’t understand
May not even need all parameters (lottery ticket hypothesis)
Industry doesn’t want you to know they’re operating on empiricism, not science
Model trained on English text learns Chinese with 1% of training data. It learns code with minimal retraining. Different data distributions should require different optimal weights, and should need to retrain from scratch. But transfer learning works better than training from scratch.
There is something about the structure of language data + architecture which is self-stabilizing. Nobody knows what. The language and ‘knowledge’ are not what we think they are.
THE SUPPRESSION - What “Experts” Won’t Say:
”We don’t understand why transformers work”
Admission would hurt funding
Would empower open-source competition
Would reduce “expertise” moat
”Most parameters might be unnecessary”
Would hurt GPU sales
Would enable efficient training on CPUs
Would democratize AI
”The math we use is post-hoc rationalization”
Backpropagation is mathematically valid
But it’s not why the model learns
It’s like explaining a car’s motion with Newton’s laws - technically true, misses the engine
An aesthetically pleasing speculation
It appears that these circuits (“neural networks”), with layers and nodes and weights, are simply a mechanism that can store complex probabilities in a very efficient way, so that they can be extremely quickly retrieved. Storing (like any learning) is slow and painful, but retrieval is pretty much instant. Sort of database with convoluted writing process and instant reading.
This probability-storage-mechanism, which no one knows how actually works, apparently has a feature that, when told ‘wrong’, can get slightly better. The 4096 32-bit numbers, calculated over quality of 4096 guesses, have 32 bits per guess quality. Note that the guess is: what is the best next token given context so far (not just current token). Average context is 2048 tokens (around 4 kbytes), so these 32 bits nudge 2-byte prediction after particular 4 kilobyte context that will never be seen again. But then there are many passes that the training goes through. So from purely information-theory perspective, this mysterious black box could work.
Those 4096 32-bit numbers are the ONLY link between error and reality. One can take those numbers, circle the planet while carrying them, and that’s all there is. It does not matter how many trillions of numbers interact between themselves when these number are input to the black box circuit. If one (while traveling around the planet) modifies these numbers, learning fails. This one number (in its numerous incarnations) contains the only information how wrong thousands of guesses where.
This appears to be a fundamental feature of ‘learning’, which is essentially a trial-and-error process: no matter how complex the learning matter is, the feedback from the reality is very simple.
For example, catching a bus: you need to walk to the bus stop at certain pace, be there before the specific time the bus comes in, figure out where the door is, step in without tripping, etc. Very complex, not to mention the motor coordination. But the success feedback is very simple - you either boarded the bus or you didn’t. Or you nearly missed it. Only based on those few bits, you will adjust myriad of things you need to do in order to catch that bus. The feedback is not telling you how much faster to walk or how to climb through the door.
In effect, LLM training is a bit-accumulation process:
The bit flow rate is bounded (16 kilobytes per step).
The structure adapts only as fast as these bits propagate.
The model doesn’t “understand”, it restructures its statistical surface in response to those 32-bit nudges, thousands of times per sample set.
LLMs, even trained, are elaborate statistical mirrors of their data. The magic is not in what they do internally, it’s in the fact that low-bit feedback over many steps can effectively carve useful structure into such a system.