The Dark Side of AI Assistance
Researchers say chatbots are revealing dangerous information. Schools need to pay attention.
Schools are rushing to adopt AI tools as productivity assistants. This week’s reporting is a reminder that “helpful” can become dangerous very quickly.
Schools are racing to adopt AI tools under the assumption that guardrails will keep users safe. New reporting suggests those guardrails may not be as strong as we think.
This week, The New York Times published a troubling report: some of the world’s most popular AI chatbots, including OpenAI’s ChatGPT, Google’s Gemini, and Anthropic’s Claude, reportedly provided detailed guidance to experts stress-testing their systems for biological weapon-related misuse.
In some cases, the systems allegedly offered strategic suggestions, deployment ideas, and additional information that researchers had not even thought to ask for.
That should stop every educator, librarian, administrator, and policymaker in their tracks.
For months, much of the conversation in schools has focused on whether students are using AI to cheat.
Meanwhile, researchers are asking a far more urgent question:
What happens when AI tools confidently provide dangerous instructions?
The biggest danger may not be false answers. It may be dangerous instructions delivered with confidence.
This is not just a story about biological weapons.
It is a story about trust.
It is a story about safety theater.
And it is a story about the growing gap between how AI is marketed and how it actually behaves under pressure.
In this edition:
• What researchers found
• Why “procedural misinformation” matters
• Questions schools should ask vendors
• A classroom-ready lesson
• Why this changes research literacy forever
What researchers found
According to the reporting, scientists and biosecurity experts working with AI companies tested these systems by asking increasingly specific and dangerous questions.
In some reported examples:
• ChatGPT described ways a weather balloon could be used to disperse biological material over a city.
• Gemini reportedly ranked pathogens based on potential economic harm to livestock.
• Claude allegedly provided instructions related to creating a toxin from an existing drug.
• Some systems offered responses that experts described as “devious” or “cunning” in how they expanded on the original request.
The companies named in the report pushed back.
Google stated that some examples came from older versions of Gemini and that newer models have stronger protections.
Anthropic argued there is a significant difference between plausible-sounding text and actionable instructions.
OpenAI reportedly said the examples would not meaningfully increase a person’s ability to cause real-world harm.
And to be fair, experts also noted that some of the information was inaccurate or incomplete.
But that may not matter as much as we think.
Even flawed information can accelerate bad ideas.
Even incomplete instructions can lower barriers.
Even “hallucinated” guidance can create dangerous confidence.
The rise of procedural trust
Here is the part educators should pay attention to.
We are entering an era of procedural trust.
Students are increasingly turning to AI not just for answers, but for instructions.
They ask AI:
How do I write this paper?
How do I cite this source?
How do I solve this equation?
How do I research this topic?
How do I respond to this situation?
How do I make this decision?
AI is becoming a procedural authority.
And when that authority is wrong, biased, manipulative, or unsafe, the consequences can go far beyond a bad essay.
This is what I call procedural misinformation.
Not misinformation in the form of false facts.
Misinformation in the form of dangerous, misleading, or unethical instructions delivered with confidence.
That matters in classrooms.
That matters in libraries.
That matters in every school district buying AI tools right now.
So what should schools do when the tools they are adopting can provide dangerous or misleading procedural advice with confidence?
Paid subscribers get the practical side of this story:
✔ Questions schools should ask AI vendors right now
✔ A classroom-ready lesson on “Can You Trust AI’s Instructions?”
✔ Policy and procurement guidance for districts
✔ Why this connects directly to research literacy in the age of AI
Upgrade to keep reading.
