Lately I’ve been thinking about the averaging itself, as a function of AI. Large language models are, by definition, engines of averaging. When queried, they find the most probable, the most representative, the most average response across the distribution of everything they’ve been trained on. This is part of why models hallucinate, a big reason why they are inherently biased, and why I believe the architecture is fundamentally flawed, though that’s another post.

As I have experimented with AI to assist in my own work, it performs well on certain tasks (organization, summarization, recipes) and poorly on others (regulatory investigation, creative ideation and evaluation, writing creatively.) In the creative writing use case it is startlingly bad, I am almost always disappointed with the outcome, and I’ve found that the time saved becomes time wasted as everything needs to be re-written. As Eve Fairbanks articulated, “nothing is quite right” in the creative output. It all comes across to me as sort of bland.

A common discussion thread as to the reason for this is that the averaging [see above: regression] that makes LLMs so good at coming up with an acceptable response most of the time is what keeps them from coming up with an exceptional response. . .ever. This is one component of a well known phenomena in machine learning known as “model collapse”, caused by something resembling the following:

1. An LLM is trained on all of human knowledge and governed with a response architecture telling it to return the statistically most relevant response to any question.

2. A human prompts an LLM for an output. Multiply this by a jillion to represent all of human usage.

3. The LLM returns a response - which on the average, is average [see above again: regression]. Multiply this by a jillion to account for all the new IG posts, LinkedIn updates, High School spanish homeworks, and college entrance exams created by these queries.

4. Update the training set with all this new “content”.

5. Human requests a new output from LLM, however this time, the response is even more average, because the training set contains it’s original data + all of the incrementally generated average responses.

6. Repeat steps 1-5 until the responses are entirely useless.

This is a known issue, in which “data quality” is a key AI workstream that many consider a significant technical blocker to improvement in output quality. Will we run out of verifiably human created training data? Probably at some point. I mean, take a look at this mind blowing chart from a ProfG Media report:

There are many ways of band-aiding this output degradation (synthetic data verification, ensemble models, etc) but that’s not the point of this post.

There’s a second-order problem that doesn’t get discussed enough. It’s not just that the outputs get blander. It’s that the thinking gets blander that’s prompting the outputs.

After enough of the above loops to develop a working relationship with consumer AI, add these steps to the sequence:

1. Human requests a new output from LLM, however this time, the request itself is blander because the human has gotten a little lazy and lost some of their critical thinking capability due to cognitive outsourcing. (Don’t take my word for it.)

2. A bland request gets an even blander output. (Bland x bland = bland²)

3. This (plus all of the AI generated “books” on Amazon) get used as the input for the next training run.

4. And so on, ad infinitum.

This is problematic, as it accelerates the already existing issue of averaging at the expense of our own critical thinking ability into a kind of self-fulfilling enfeeblement cascade. I call this accelerated phenomena, “recursive enshittification”, and while not a perfect analog to the original “enshittifcation”, it was too hard a coinage to resist.

The brain muscle that produces original thought doesn’t get used. Over time, it gets harder to use. When something is hard, we tend to avoid it, and what started as apathy becomes inability. This didn’t start with LLMs, it started maybe with rock music, then TV, then probably with video games, then definitely with cell phones and social media. When I moved to LA, I frequently got lost if I didn’t have my Thomas Guide, but now we don’t go anywhere without a GPS. LLMs are a horizontal tech that introduces information ease across many domains simultaneously and they are on loudspeaker. I mean, I’m concerned about my child’s ability to navigate the world - thank the gods he reads!

Understand, this isn’t a diatribe against LLMs. On the contrary, their existence, influence, and impact is fascinating to me - fascinating enough to get me to take classes and try to write about it. But I do have concerns about our overall ability to contend with distraction, focus critically on a task, produce original thought response, and grow our own understanding as a result. It’s called learning (dummy) and it takes actual work - work which is being relinquished very quickly to an easier faster average.

What does this work look like? For me, in this instance you’re reading, it looked like this:

1. Have an idea - in this instance a phrase (recursive enshittifcation) - and consult with an LLM about how to articulate it.

2. Be disappointed in the result and feel lazy and shitty about beginning with step 1 at all.

3. Stew on it for a few days, read some relevant research, essays, blog posts, and take various stabs at writing about it - away from my LLMs.

4. Spend time to finally flesh out my own text and feel like I actually accomplished something. Bonus: I’ve also internalized the thought stream now through my own reading, thinking and writing, and could recall and defend it at will. This learning thing is magical!

If you eschew my process, here are a few focused approaches that may work for you:

1. Use your LLM for a sparring partner

2. Engage in chain-of-thought prompting to build your skepticism muscle with regard to LLM outputs

3. Use an LLM to gather resources for you, not as the resource itself, and then actually read them rather than getting AI summaries.

4. Come up with your own approach - remember. . .it takes work!

I’ve been thinking about something that seems obvious once you say it out loud, but that few parties in the AI product space appear to be treating as a priority.

Current large language models do two things at once. They find an answer to your query and they render it. Sometimes they find the right answer, sometimes they don’t. The LLM architecture is probabilistic. It isn’t designed to deliver truth. It’s designed to statistically deliver the most probable response to a question. (Why this results in six-fingered hands I do not know. Perhaps we should have a chat bot named Inigo Montoya. . .)

Then they render the answer with confidence and fluency designed to make the answer sound good and plausible and truthful. They do this part (the good, plausible, truthful part) with almost perfect success. I’m not left handed either!

Those are not the same objective, and treating them as if they are produces a system that is by design simultaneously overconfident and underinvested in accuracy. I suspect one is likely compromising the other. When those two processes are enmeshed, and fluency is influencing the confidence with which an uncertain answer gets rendered, you end up with something that feels reliable and isn’t.

How many intelligent, technically literate people you know have told you they regularly query one model and fact-check it against a second? Probably at least a few. Think about that for a moment: the workaround for a model’s unreliability is a second model. That’s not a solution. It’s a bug, not a feature.

What if the architecture were designed from scratch to be truthful and finding the answer and rendering it were treated as separate problems? When you’re finding the answer, the only thing that should matter is whether it’s true. Accuracy. Calibration. When you’re delivering the answer, you can be as artful as you want - or just give me bullet points - but if you don’t then give me an honest representation of uncertainty.

I’ve read about architectural approaches that try to address this - systems that separate a retrieval or reasoning layer from a language generation layer. They exist. They have their own problems. But the instinct behind them is right: truth-finding and truth-telling are not the same cognitive act, and they probably shouldn’t be the same computational act either.

So why isn’t this a priority of the AI product ecosystem? It’s about incentives Mr. Munger! (Just look at the economics.)

The companies building frontier AI models are not primarily research institutions with a public interest mandate. They are, or are rapidly becoming, the most valuable commercial enterprises in human history. Microsoft, Google, Amazon, Meta, and Nvidia now account for a disproportionate share of the entire US stock market - somewhere in the vicinity of a third of total market cap concentrated in a handful of companies whose fortunes are increasingly tied to the outcome of the AI race. The financial stakes of that race are so large that they have ceased to function as incentives and have become something closer to a gravitational field. Everything bends toward them. P.S. - tech’s share of US market cap doesn’t yet (directly) include OpenAI, Anthropic, and SpaceX.

Nvidia sits at the center of this in a way that deserves more attention than it gets: it’s a monopoly on the hardware that makes frontier AI development possible. Every major lab - OpenAI, Anthropic, Google DeepMind, Meta AI - depends on Nvidia GPUs to train their models. Nvidia knows this, and prices accordingly. The result is that the infrastructure layer of the AI race is controlled by a single vendor with every incentive to keep the race going as fast as possible and no meaningful incentive to pump the brakes. When the snow shovel supplier has this kind of stranglehold on the market, the snow shovels don’t get less expensive when the sun comes out.

Then there are the IPOs: OpenAI, Anthropic, and SpaceX all moving toward public offerings. The moment these companies become publicly traded, their obligations to AI safety - already in tension with their obligations to investors - will be formalized in a way that leaves very little ambiguity about which obligation takes priority. Public markets are not patient. They do not reward a company for spending an extra year on alignment research before shipping. They reward growth, revenue, and the appearance of competitive momentum. The CEO who slows down to get it right will answer for that decision in an earnings call. The CEO who ships fast and breaks things will be celebrated right up until something breaks badly enough to make the front page. (Note: Anthropic is the only one of these who, as a PBC, has made a significant structural commitment to safety. I’ve included them here to illustrate the market argument, not the safety argument - though by my own reckoning they are intertwined.)

The incentive here is not truth or long-term wellbeing of the people who will depend on these systems. The incentive is market position, and market position in a winner-take-all race goes to whoever moves fastest. In that environment, investment in accuracy and calibration is not just underprioritized - it is structurally penalized. A model that says “I don’t know” loses to a model that gives you a clean paragraph or a dainty spreadsheet with a carefully selected color palette. A company that pauses to audit its systems or rethink its product design loses ground to a company that ships. And when every player faces the same logic simultaneously, it’s a game of hot-rod chicken where nobody can afford to blink.

When you reduce this dynamic from its massive scale down to a 1:1 interaction with an AI, remember this. It’s not invested in giving you the truth. Its interest is in giving you something fast and clean and designed to exceed the quality expectations of your own first draft thinking. (In my case, this is a low bar.) I’m not suggesting you shouldn’t use Claude to gussy up your spreadsheet, but one should be wary of such jazz hands in a query of any consequential nature. I have learned (the hard way sometimes) to fact-check even the appearance of facts. It’s good practice anyway!

As always, James is human and can make mistakes. Please double check responses.

Side note: Nvidia’s success is not a market failure in the technical sense. It is the market working exactly as designed. The problem is that the market was not designed with this particular technology in mind, and nobody has yet built the governance infrastructure capable of changing the rules of the game.

This is one of the many reasons why I’ll continue my coursework and my own deep dive for the truth, starting with BlueDot’s Frontier AI Governance course this summer. Hope to see you there!