The Question
Can AI help me learn by asking questions instead of providing answers?
Most people use AI as an answer machine. In this experiment, I wanted to test something different.
Instead of asking ChatGPT to teach me a topic, I asked it to become a Socratic tutor whose goal was not to explain more information, but to discover whether I actually understood what I had already read. The topic was the Illusion of Explanatory Depth.
The Setup
Topic: Illusion of Explanatory Depth
Models: ChatGPT
Process:
- Ask ChatGPT for a short explanation of the topic.
- Read the explanation.
- Switch ChatGPT into Socratic Tutor mode.
- Allow ChatGPT to ask one question at a time.
- Follow the conversation wherever it leads.
The prompt that started the experiment:
Now stop explaining.
Act as a Socratic tutor instead.
Your goal is not to teach me more information but to discover whether I truly understand the summary I just read.
Ask me one question at a time.
How to Read This Experiment
The quoted sections below are selected excerpts from the conversations.
The shaded note boxes are my observations as the human in the loop.
They are not summaries of what the AI said.
They are the places where I noticed a pattern, a shift, or a question worth carrying forward.
The Conversations
Model: ChatGPT
Excerpt 1: Understanding vs Recognition
ChatGPT:
In your own words, what is the difference between recognizing something and understanding how it works?
My Response:
Recognizing is being able to identify an object or a function. Understanding is to be able to describe the underlying functions and mechanisms.
PixelPia Note
This was where I expected the conversation to stay.
At this point it felt like a simple comprehension check. I assumed I understood the topic fairly well after reading the summary.
I was wrong.
Excerpt 2: The Real Problem
ChatGPT:
Suppose a person knows only part of how something works but believes they understand it completely.
Which part of the illusion is more important:
• the fact that their knowledge is incomplete, or
• the fact that they misjudge the completeness of their knowledge?
Why?
My Response:
The fact that they misjudge the completeness of their knowledge is more important because then you don’t understand or even think that there is more to learn.
PixelPia Note
This was the first moment where the conversation moved beyond the topic itself.
Incomplete knowledge is normal. None of us understand everything.
What interested me was the distinction between not knowing and not realizing that you don’t know.
The first creates an opportunity to learn.
The second can stop learning before it begins.
As the conversation continued, I started realizing that this wasn’t only about psychology. It was also about how we use AI. A convincing answer can make us feel informed, but feeling informed and understanding something deeply are not necessarily the same thing.
Core Question
Is the real problem a lack of knowledge, or a lack of awareness of what we don’t know?
Excerpt 3: AI Enters the Conversation
ChatGPT
Why might having easy access to other people’s knowledge make it easier to overestimate your own understanding?
My Response
In the same way my interaction with AI can make me feel that I understand something from another source, without testing my own knowledge and understanding. If one expert gives me an answer that feels right to me I accept it.
PixelPia Note
This was the moment where the conversation stopped feeling like a discussion about the Illusion of Explanatory Depth and started feeling like a discussion about AI.
One reason AI is so useful is that it can make complex ideas feel accessible. A good explanation removes friction. It helps us move from confusion to clarity very quickly.
The problem is that clarity and understanding are not necessarily the same thing.
When an explanation feels right, it is tempting to assume that learning has occurred. But all I had really demonstrated at this point was that I could follow the explanation. I hadn’t yet shown that I could reproduce it, apply it, or explain it myself.
That distinction became one of the most important ideas in the entire experiment.
Core Question
When does access to information start feeling like understanding?
Excerpt 4: The Question That Changed the Experiment
ChatGPT
Suppose you read a clear explanation from an expert and it makes perfect sense while you’re reading it.
What additional test could you perform on yourself to find out whether the understanding now exists in your mind rather than merely seeming familiar because you just encountered the explanation?
My Response
To create my own explanation maybe in writing.
PixelPia Note
This was the moment that stopped me.
Up until this point, the conversation had been about the Illusion of Explanatory Depth. Suddenly it became about learning itself.
Most of us have experienced the feeling of understanding something while we are reading it. The explanation makes sense. The logic is clear. Everything seems to fit together.
But that feeling can be deceptive.
Recognizing an explanation is not the same as generating one. Following a line of reasoning is not the same as being able to recreate it from memory. Understanding often reveals itself only when we try to explain, apply, or teach an idea without the original source in front of us.
The question forced me to shift from asking, “Do I understand this explanation?” to asking, “What evidence do I have that I understand it?”
That is a very different question.
Core Question
How can I tell whether understanding exists in my mind instead of simply feeling familiar?
Excerpt 5: The Deepest Question
ChatGPT
Imagine someone spends hours talking with AI about a topic. The explanations are clear, the conversation feels productive, and the person can follow every step while reading.
Before they ever try to explain the topic themselves, solve a new problem, or teach it to someone else:
What evidence do they actually have that the understanding exists in their mind rather than in the conversation they just had?
PixelPia Note
This was the question I chose to leave open at the end of the video.
Not because I couldn’t answer it, but because it felt larger than the experiment itself.
The question applies to almost every interaction we have with AI. We can spend an hour having an insightful conversation. We can follow every step of the reasoning. We can feel engaged, productive, and even inspired.
But what remains after the conversation ends?
If I close the chat window and cannot explain the idea in my own words, apply it to a new situation, or teach it to someone else, then what exactly have I learned?
The more I thought about it, the more I realized that this experiment wasn’t really about ChatGPT. It was about the difference between participating in understanding and possessing understanding.
AI may be able to help us think, but it can also create the illusion that the thinking happened somewhere outside of us.
Core Question
What evidence do I actually have that learning has occurred?
Conclusion
This experiment began as a test of the Illusion of Explanatory Depth.
It ended as a reflection on learning.
The most valuable part of the conversation was not the information ChatGPT provided.
It was the questions.
For one experiment, ChatGPT stopped being an answer machine.
It became something closer to a tutor.
Not because it taught me.
Because it forced me to think.
Questions This Experiment Woke
- How often do I mistake familiarity for understanding?
- What is the difference between following an explanation and understanding it?
- What tests actually reveal understanding?
- Could other AI models guide this kind of conversation differently?
- What role should questioning play in AI-assisted learning?
What Didn’t Make the Video
The conversation continued well beyond the excerpts shown here.
Some of the most interesting moments involved expertise, self-assessment, and whether experts can experience the Illusion of Explanatory Depth.
The deeper we went, the less the conversation seemed to be about psychology.
And the more it seemed to be about learning itself.