Into the engine room: bagging, boosting, attention and why it all happened at once

In the previous article we untangled the acronym soup on the surface, what AI, ML, DL and NLP actually mean, and what an LLM does at its core. This article goes one level deeper: how do these systems actually make their decisions, and why did the last few years feel like everything exploded at once?

Asking several experts: bagging vs. boosting

One of the older, more robust ML tricks is to not trust a single model, but build many small models and combine their answers. There are two very different ways to do that, and the difference between them says a lot about how each approach thinks about error and uncertainty.

Bagging is like asking a hundred independent experts the same question and going with the majority vote. Each expert (in practice a small decision tree) is trained on a slightly different sample of the data, with no knowledge of what the other experts are doing. The point isn't that each individual expert has to be good. It's that their individual mistakes tend to cancel out once the votes are combined. Random Forest, one of the most widely used ML models in practice, is essentially exactly this: hundreds of decision trees voting together.

Boosting works the opposite way. Instead of a hundred independent experts, you build them in a chain, one after another, where each new expert is specifically tasked with focusing on what the previous experts got wrong. The first model does its best, the second is trained almost exclusively on the cases the first one missed, the third focuses on what the first two still miss, and so on. The result is often more accurate than bagging, but because the models are built in sequence, one at a time based on the previous errors, boosting is more sensitive to noise in the data and easier to overfit if you're not careful. XGBoost, which has won a long list of data science competitions, is a boosting model.

The short version: bagging reduces variance through independent voting, boosting reduces bias by actively chasing previous mistakes. Neither of them is an LLM or even deep learning, but they remain the models most commonly used in practice for structured business data (credit scoring, churn prediction, pricing), precisely because they're fast, robust, and relatively easy to explain to an auditor.

What attention actually does

Before 2017, text was essentially read word by word, in a long chain, where each word only influenced the next step through whatever hadn't already been slightly "forgotten" along the way. It worked, but it got worse the longer the sentence was, much like a game of telephone where the message degrades with every person it passes through.

Attention solved that with a simple but powerful idea: let every word in a sentence "look at" every other word at the same time, and calculate how relevant each of them is for understanding this particular word. The word "it" in the sentence "the dog chased the cat because it was angry" has to weigh both "dog" and "cat" to figure out who "it" refers to, and attention is the mechanism that lets the model make exactly that trade-off, for every word, simultaneously, instead of reading in a straight line.

That mechanism, described in the now-famous paper "Attention Is All You Need", is what underlies the transformer architecture, and by extension, practically every LLM that exists today.

Why did everything explode right now?

The transformer architecture was introduced back in 2017, but it took several years before it led to tools like ChatGPT. The reason isn't a single stroke of genius, but three things that happened to fall into place around the same time:

• The architecture: transformers with attention made it possible to train on much larger datasets much more efficiently than previous architectures, since the calculations for each word can be done in parallel instead of in sequence.
• The data: by this point, the internet had grown into an almost unimaginable amount of freely available text, exactly the kind of raw data that deep learning (as we established in the previous article) can learn from without anyone having to label it by hand.
• The compute: GPUs, originally built to render graphics in video games, turned out to be extremely good at exactly the kind of parallel calculations transformers require, and became cheap enough at scale to train models with hundreds of billions of parameters.

None of the three would have been enough on their own. Without attention, the large datasets and compute couldn't have been used effectively. Without internet-scale text data, the architecture would have had nothing to learn from. And without cheap, parallelizable compute, the whole thing would have been economically impossible to train at the required scale. That's why it felt like an explosion rather than a gradual improvement: three curves that all crossed a threshold at roughly the same time.