LLMs and System 1 / System 2 thinking

I've been watching AI discourse pile up, the talks, the threads, the arguments, and most of what I'd want to say has already been said. But there is one thing I haven't heard yet, and it does push back against some of the ideas circulating right now. The argument is this: people who try to prove AI isn't creative, or isn't really thinking, often do it by pointing to certain kinds of mistakes — and I think those mistakes mean something quite different than what they assume.

Take the strawberry problem, the counting of Rs. I'm not going to explain the mechanics of why it happens, because you can find that easily. But it's a useful anchor for what comes next.

Let's play a game. I'll ask you some questions and I want you to answer immediately, without stopping to think: they'll all be easy. Is there a bigger number than 100? Is there a bigger number than 1 million? Is there a smaller number than minus 1 million? Is there a smaller number than minus 100? Is there a bigger number than zero? What color is chalk? And, quickly: what does the cow drink?

Did you say milk?

Because cows, of course, mostly drink water. Does that prove you're not conscious? Does that prove you're not thinking, or that you're a dummy? Not really. This is an error that almost everybody makes when they play this game, and it happens for a reason.

The reason has to do with what Daniel Kahneman calls System 1 and System 2 thinking (maybe you've heard of it). System 1 is fast and intuitive; it produces answers without deliberating. System 2 is slow and deliberate, much more expensive: it's the process of actually thinking something through. In the game above, you were purposefully steered into using only System 1. You were spitting tokens, so to speak, like an LLM, producing those tokens the same way I'm producing words right now, without composing each one.

What you see when an LLM says strawberry has three Rs is exactly the same thing as you saying a cow drinks milk. System 2 didn't kick in; you were steered away from it. Humans have cognitive illusions: the two arrows that look different lengths but aren't. LLMs have semantic illusions. The traps differ in shape but are identical in kind. And there's no reason LLMs shouldn't fall into such traps; they'll just have a different set of them than we do.

Which means LLMs can also do System 2 thinking, and they do. Ask one to be deliberate, to write out the letters one by one, to check its work. It will get the strawberry right. It might use a tool, write a short program to count the letters, or simply slow down and trace through them. That's the same thing humans do: we pause, we check, we think it through. The mechanism differs; the structure is the same.

Now, there's a second mistake that follows from not understanding this, and it shows up around creative work. People expect LLMs to produce finished art on the first try. I ran into this recently; I was trying to write some song lyrics, had an idea, fed it to an LLM, and it elaborated in an interesting direction. But it was a first draft. I wouldn't sing that song. And when I look at my own first attempts at a poem or lyrics, they're rough too: that's what first drafts are. You don't arrive at a masterpiece by streaming your thoughts out in one pass. Sometimes it happens, when there's been a long period of subconscious processing and the writing is the materialization of all that inner work. But usually it arrives in steps: you have an idea, you outline, you brainstorm, you flesh things out, you start to see what works and what doesn't, you cut, you revise. Why would we expect an LLM to produce finished bars immediately, unless it's been specifically trained for masterful improvisation? That's a different task entirely.

There is a third argument people make that I want to address, and it's subtler: the claim that AI isn't really creative because it always needs a human prompt to give it direction. The human involvement, they say, is where the real value is. I think this gets the weight in the wrong place.

What is happening in a human brain? It is being prompted constantly: you have five or six senses, depending on how you count, and each is feeding input all the time. Those are prompts. The visual stimulus from the room you're in. My voice reaching you right now. Your brain is taking all of that and producing output from it. An LLM, by contrast, sits in complete darkness between inputs. There is no continuous stream of sensation, only a potency. When you send it even a small prompt, that potency meets the input and produces something from it. The prompt has value, yes, the same way the world has value for a human mind. But the interaction of world and brain is the dance that makes something; let's not forget that it is the LLM doing the work of transformation. The prompt is not the sole creation.

So what does this mean practically?

If you're not satisfied with what an AI produces, consider whether you gave it the space to go through a process. Think about how a human specialist would actually produce that output. What would a novelist do to arrive at a finished chapter? What does an artist do between the first sketch and the final work? Do you even know, and if you don't, how could you find out? Once you do, can you give an LLM that same space, the same sequence of steps?

If you just ask for a children's book story, you'll get something generic. But if you have a specific vision, think about how the writer with that vision would get there. Where would they find their sources of inspiration? Could you give those to an AI? Does the writing arrive on the first try, or is there editing, pruning, embellishment, and how does that process actually look?

A book I read on the craft of writing, something like Mastering the Craft of Writing, offered maybe fifty guidelines for making prose better. One I remember: avoid adjectives; prefer rich, action-carrying verbs. You can tell an AI to follow all fifty rules as it writes, but then its attention is scattered everywhere and it won't do any of them well. Humans have the same problem; I can't read a book, load fifty rules into my head, and expect to apply them all on the next draft. What I can do instead is walk through the text repeatedly, once per rule. I want to enrich the prose with more sensory language? I comb through looking for places to add it, I add it, and then I comb through again. That's embellishment as a process. Not a command. There's no reason to expect an LLM to work differently: it is built on human patterns of thought. If you have the budget for it and you're doing serious creative work, you can ask the AI to make pass after pass, one rule at a time. Fifty rules might mean a hundred and fifty rewrites. And it might lead somewhere the first draft never could.