AI makes you faster, but does it make you better? The hidden cost of AI coding assistance

Contents#

• Introduction
• The Experiment
  • The Two Tasks
• The Stark Results
  • The Debugging Gap
• Six Patterns of AI Usage
  • Low-Scoring Patterns (24-39% quiz scores)
  • High-Scoring Patterns (65-86% quiz scores)
• What Participants Said
• The Pair Programming Connection
  • Why Pair Programming Preserves Learning
  • The Right Balance: AI + Pair Programming
• Implications for the Industry
• Practical Takeaways
• Conclusion

Introduction#

A new research paper from Anthropic has dropped, and its findings should give every developer pause. The study, titled "How AI Impacts Skill Formation" by Judy Hanwen Shen and Alex Tamkin, investigates what happens when developers use AI to learn new programming concepts.

The headline finding? AI assistance can significantly impair your ability to actually learn.

"AI use impairs conceptual understanding, code reading, and debugging abilities, without delivering significant efficiency gains on average."

This isn't just academic speculation; it's a rigorous randomized experiment that measured real learning outcomes. And the implications for how we integrate AI into our workflows are profound.

The Experiment#

The researchers designed a clever study around learning a new Python library called Trio, which handles asynchronous programming. They chose this library specifically because:

1. It's relatively unknown (compared to asyncio)
2. It involves genuinely new concepts like structured concurrency
3. It's explicitly designed to be easy to learn-making it ideal for a learning experiment

Participants were professional developers with 1-7+ years of experience, randomly assigned to either:

• Treatment group: Could use an AI assistant (GPT-4o) to complete coding tasks
• Control group: Had to complete tasks using only documentation and web search

The Two Tasks#

Participants had to complete two specific coding tasks using the Trio library:

• Task 1: Concurrent Execution - Write a timer that prints every passing second while other functions run concurrently. This task introduces the core concepts of nurseries, starting tasks, and running functions concurrently in Trio.

• Task 2: Error Handling with Memory Channels - Implement a record retrieval function that can handle missing record errors. This task introduces concepts like error handling and memory channels to store results.

After completing these tasks with a 35-minute limit, all participants took a comprehensive quiz covering conceptual understanding, code reading, and debugging.

The Stark Results#

The AI group scored significantly worse—and wasn't even faster at completing the tasks.

"We find that using AI assistance to complete tasks that involve this new library resulted in a reduction in the evaluation score by 17% or two grade points... Meanwhile, we did not find a statistically significant acceleration in completion time with AI assistance."

The Debugging Gap#

When broken down by question type, the biggest gap was in debugging skills. Why? The AI group encountered far fewer errors:

(Median errors per participant)

The control group had to grapple with RuntimeWarning when a coroutine was never awaited, or TypeError when passing the wrong type to a Trio function. These struggles forced deeper understanding—errors were learning opportunities.

"We attribute the gains in skill development of the control group to the process of encountering and subsequently resolving errors independently."

Six Patterns of AI Usage#

One of the most valuable contributions of this research is the identification of six distinct AI interaction patterns. Through qualitative analysis of screen recordings, the researchers categorized how developers actually use AI assistance:

Low-Scoring Patterns (24-39% quiz scores)#

1. AI Delegation: Only asked AI to generate code, pasted it directly. Fastest completion of 19.5 min but worst learning (39% quiz score).

2. Progressive AI Reliance: Started with questions, then delegated everything to AI. Scored poorly because they never mastered Task 2 concepts.

3. Iterative AI Debugging: Used AI repeatedly to debug and verify code. High query count but relied on AI to solve problems rather than understanding them.

High-Scoring Patterns (65-86% quiz scores)#

4. Conceptual Inquiry: Only asked conceptual questions, wrote code independently. Fastest among high scorers (24 min) with strong learning (86% quiz score).

5. Generation-Then-Comprehension: Generated code with AI, then asked follow-up questions to understand it. Not fast, but demonstrated high understanding.

6. Hybrid Code-Explanation: Asked for code generation along with explanations. Reading and understanding the explanations took more time but preserved learning.

The pattern is clear: cognitive engagement is what separates those who learn from those who don't.

"These three patterns of interaction with AI, which resulted in higher scores in our skill evaluation, involve more cognitive effort and independent thinking."

What Participants Said#

The qualitative feedback from participants tells the story:

From the AI group:

"By using the AI assistant, I feel like I got lazy. I didn't read the Trio library intro and code examples as closely as I would have otherwise."

"I wish I'd taken the time to understand the explanations from [the AI] a bit more!"

"I would not have minded going into a little more depth with the assistant to really understand and prove out the details of the trio library. I feel like I got a pretty good overview but there are still a lot of gaps in my understanding."

From the control group:

"It was fun to learn about asynchronous programming, which I had not encountered before."

"The programming tasks were very fun and did a good job of helping me understand how Trio works despite never having used it before."

The control group actually reported higher self-reported learning and comparable enjoyment levels, despite finding the tasks more difficult.

The Pair Programming Connection#

Here's where this research becomes particularly relevant for us at Hopp. The paper explicitly identifies human assistance as an important counterfactual they didn't study:

"We do not include the counterfactual of how skill formation would be impacted by receiving assistance from humans. Since human assistance and feedback takes place in diverse settings (e.g., classroom, pair programming, code review), future work can compare the effect of feedback from AI vs humans in all these settings on skill formation."

This is a crucial gap—and one where pair programming has a natural advantage.

Why Pair Programming Preserves Learning#

Looking at the high-scoring AI usage patterns, they all share one key characteristic: sustained cognitive engagement. This is exactly what pair programming enforces by design.

1. The Navigator/Driver Model Forces Engagement

In traditional pair programming, one person drives (writes code) while the other navigates (reviews, thinks ahead, catches errors). Neither role allows you to disengage. The navigator can't just paste AI output—they have to understand the code well enough to provide useful guidance.

2. You Have to Explain Your Thinking

When you pair with a human, you're constantly articulating your reasoning. "I'm going to use a nursery here because..." This verbalization of thought process is itself a learning mechanism. Research on the "protégé effect" shows that teaching others improves your own understanding.

3. Errors Become Collaborative Learning Moments

When you hit a RuntimeWarning: coroutine was never awaited while pairing, you don't just ask AI to fix it—you work through it together. Your pair might say, "I think we need an await here because..." and you both develop the mental model.

4. Social Accountability Prevents Shortcutting

With AI, it's tempting to just accept the generated code and move on. With a human pair, there's social pressure to actually understand what you're writing. You can't just say "the AI wrote this, I don't know how it works."

The Right Balance: AI + Pair Programming#

The solution isn't to abandon AI. It is to use it wisely. The research shows that some AI usage patterns preserve learning. We can combine the best of both:

• Use AI for conceptual questions while pairing: "Hey, can you explain when I'd use a memory channel vs a queue in Trio?" This maps to the high-scoring "Conceptual Inquiry" pattern.

• Generate code, then explain it to your pair: If you do use AI to generate code, take the time to walk through it with your partner. This maps to "Generation-Then-Comprehension."

• Use AI output as a starting point for discussion: Instead of just accepting AI suggestions, treat them as conversation starters. "The AI suggested this approach—what do you think?"

The key insight from the research is that cognitive effort is not optional for learning. Pair programming naturally enforces this effort, while unrestricted AI access can let you skip it.

Implications for the Industry#

The researchers are careful to note limitations; this was a single task over a short time period. But the implications are significant:

"Together, our results suggest that the aggressive incorporation of AI into the workplace can have negative impacts on the professional development of workers if they do not remain cognitively engaged."

For junior developers especially, this is a critical warning. The developers who benefit most from AI productivity gains are also the ones who have the most to lose in skill development.

"While studies find that junior or less experienced developers experience greater productivity uplift from using AI, these very same workers should be quickly developing new skills in the workplace."

The paper concludes with a call for intentionality:

"Participants in the new AI economy must care not only about productivity gains from AI but also the long-term sustainability of expertise development amid the proliferation of new AI tools."

Practical Takeaways#

If you want to actually learn while using AI assistance, the research suggests:

1. Ask conceptual questions, not just "write this code": The "Conceptual Inquiry" pattern had the highest quiz scores (86%) while still being efficient.

2. If you generate code, take time to understand it: Don't just paste and move on. Ask follow-up questions. Walk through the code line by line.

3. Embrace errors as learning opportunities: Resist the urge to immediately ask AI to fix your errors. Struggle with them first.

4. Pair program for learning-critical tasks: When you need to genuinely learn something new, consider pairing with a colleague instead of (or in addition to) using AI.

5. Be intentional about when you optimize for speed vs. learning: Sometimes you just need to ship. But don't let every task become an AI delegation task, especially early in your career.

Conclusion#

AI coding assistants are powerful tools, and they're here to stay. But this research is a reminder that power tools require care. A chainsaw can clear a forest in hours; but it can also take your leg off if you're not paying attention.

The developers who will thrive in the AI era are those who learn to use AI as a collaborator rather than a crutch. And perhaps the best training for that is the original form of collaborative coding: pair programming with another human, where cognitive engagement isn't just encouraged—it's required.

The full paper "How AI Impacts Skill Formation" by Judy Hanwen Shen and Alex Tamkin is available on arXiv. The annotated research data is available on GitHub.