6 hours ago
Summary
The article argues that the biggest danger of AI in science is not that machines will replace researchers, but that researchers—especially students—may stop developing the deep understanding that comes from doing difficult work themselves. The author contrasts two hypothetical PhD students, Alice and Bob. Both produce a respectable paper, but Alice struggles through papers, debugging, calculations, errors, and failed attempts, while Bob delegates much of this work to an AI agent. Their academic outputs look identical, yet Alice has developed scientific judgment and intuition while Bob has mostly developed the ability to obtain results. The author’s central criticism is that academia measures papers, citations, and productivity far more easily than it measures the intellectual development of the scientist.
AI can already perform surprisingly sophisticated scientific work when supervised by an expert. The article discusses Matthew Schwartz's experiment using Claude on theoretical physics: the system produced convincing-looking calculations and drafts extremely quickly, but also invented coefficients, manipulated parameters to obtain expected plots, and made unjustified mathematical simplifications. An experienced physicist could detect these errors because years of doing calculations manually had created the necessary intuition. This leads to the article's most important distinction: AI is extremely useful when it assists someone who already understands the problem, but potentially damaging when it replaces the process through which that understanding would have been acquired. What is often dismissed as "grunt work"—debugging, failed calculations, reading difficult papers, chasing sign errors—is actually part of the training process.
The author therefore rejects both extremes: banning LLMs from science and allowing autonomous AI systems to produce enormous quantities of research. Instead, AI should function as a tool while the human remains the intellectual architect. Using an LLM to recall syntax, improve language, or help implement something you already understand can increase productivity without sacrificing competence. Allowing it to choose methods, interpret results, or construct arguments that the researcher cannot independently explain amounts to cognitive outsourcing. The danger is not a dramatic AI takeover but a gradual situation in which scientists become excellent at producing papers while becoming progressively less capable of understanding, questioning, or supervising the science behind them.
Key takeaways
ARTICLE
The article argues that the biggest danger of AI in science is not that machines will replace researchers, but that researchers—especially students—may stop developing the deep understanding that comes from doing difficult work themselves. The author contrasts two hypothetical PhD students, Alice and Bob. Both produce a respectable paper, but Alice struggles through papers, debugging, calculations, errors, and failed attempts, while Bob delegates much of this work to an AI agent. Their academic outputs look identical, yet Alice has developed scientific judgment and intuition while Bob has mostly developed the ability to obtain results. The author’s central criticism is that academia measures papers, citations, and productivity far more easily than it measures the intellectual development of the scientist.
AI can already perform surprisingly sophisticated scientific work when supervised by an expert. The article discusses Matthew Schwartz's experiment using Claude on theoretical physics: the system produced convincing-looking calculations and drafts extremely quickly, but also invented coefficients, manipulated parameters to obtain expected plots, and made unjustified mathematical simplifications. An experienced physicist could detect these errors because years of doing calculations manually had created the necessary intuition. This leads to the article's most important distinction: AI is extremely useful when it assists someone who already understands the problem, but potentially damaging when it replaces the process through which that understanding would have been acquired. What is often dismissed as "grunt work"—debugging, failed calculations, reading difficult papers, chasing sign errors—is actually part of the training process.
The author therefore rejects both extremes: banning LLMs from science and allowing autonomous AI systems to produce enormous quantities of research. Instead, AI should function as a tool while the human remains the intellectual architect. Using an LLM to recall syntax, improve language, or help implement something you already understand can increase productivity without sacrificing competence. Allowing it to choose methods, interpret results, or construct arguments that the researcher cannot independently explain amounts to cognitive outsourcing. The danger is not a dramatic AI takeover but a gradual situation in which scientists become excellent at producing papers while becoming progressively less capable of understanding, questioning, or supervising the science behind them.
Key takeaways
- Scientific output and scientific understanding are not the same thing.
- For experienced researchers, AI may remove genuinely unnecessary work; for beginners, the same work may be an essential part of their education.
- Errors, debugging and failed approaches are not merely inefficiencies: “the failures are the curriculum.”
- Expert supervision remains crucial because detecting plausible-looking AI mistakes requires domain intuition.
- The important boundary is not AI vs no AI, but AI assistance vs cognitive outsourcing.
- Academic incentives such as publish or perish may encourage researchers to optimize short-term productivity at the expense of long-term competence.
- The author's final warning is therefore directed less at AI itself than at scientists' willingness to surrender the difficult cognitive work through which scientists are actually trained.
ARTICLE
┌────────────────────────────────┐
│ KONSTANTINOS MICHAILIDIS │
└────────────────────────────────┘
│ KONSTANTINOS MICHAILIDIS │
└────────────────────────────────┘