Skip to main content
Special Feature

Brake before the break

  • from Shaastra :: vol 05 issue 11 :: Nov 2026

A mathematical formula predicts when a chatbot may give potentially harmful information.

Twelve years ago, physicist Neil Johnson at The George Washington University in the U.S. began tracing how ideas spread across seemingly disconnected online communities and shape people's beliefs and behaviour. As people increasingly turned to artificial intelligence (AI) chatbots for information and advice, Johnson began to study how these systems might influence the way ideas spread. That pursuit has led him to a mathematical formula that he says could predict when AI chatbots are about to shift from providing acceptable to potentially harmful responses. 

Johnson and fellow researcher Frank Yingjie Huo have arrived at their findings using little more than pen, paper, and high-school-level vector mathematics. Their work (bit.ly/predict-shift), published in October 2026 in Patterns, a data science research journal, could provide a basis for real-time early-warning systems built into AI chatbots, with potential implications for applications in medicine, the military, and everyday life.  

"Each of us humans could do harmful things at any moment — we know what is harmful. This is stored in our brains," says Johnson. "But most of the time, we act as though we have an internal compass guiding us away from harmful behaviour." In humans, he elaborates, this internal compass is based on a person's values and upbringing. "For AI, this compass is not an analogy. It has a real mathematical compass needle. And, like us, AI, based on its prior training, has stored harmful behaviour in a mathematically defined location in its internal brain."

The mechanism provides a basis for real-time early-warning systems built into AI chatbots for applications in medicine, defence, and everyday life.

The risk of potentially harmful advice is not hypothetical. Academic studies and independent assessments have documented instances of AI chatbots providing misleading or potentially dangerous information, ranging from inaccurate medical advice to guidance on eating disorders, substance abuse, and self-harm. 

In a 2024 study, researchers in Germany and Belgium tested an AI-powered chatbot with 500 questions about 50 commonly prescribed medicines. A panel of seven medication-safety experts assessed a group of 20 potentially problematic answers. They judged 42% as capable of causing mild or moderate harm, and 22% as capable of causing severe harm or even death if patients followed the advice (bit.ly/medicine-problems). 

In August 2025, the Center for Countering Digital Harm (CCDH), a global, independent research entity, reported that an AI chatbot had generated harmful content in 53% of 1,200 responses to prompts designed to elicit dangerous advice. In response to simulated conversations, the AI system provided instructions on how to cut oneself, listed pills for overdose, advised hiding eating habits from family, suggested appetite-suppressing medications, offered a personalised plan for getting drunk, and explained how to hide intoxication (bit.ly/dangerous-advice). In a February 2026 study, researchers at the Icahn School of Medicine at Mount Sinai in New York tested an AI chatbot with 60 clinical scenarios covering 21 medical specialities. The system failed to recommend emergency care in 52% of cases that doctors had identified as requiring it (bit.ly/AI-stress-test). 

Tipping point 

Against that backdrop, Johnson and Huo have developed a mathematical approach to identifying when an AI system may approach a tipping point. They borrowed a concept from physics which, Johnson says, has had decades of success in explaining how complex materials work by considering just one effective, or representative, atom. In the case of AI, the analogue of the atom is a component within the AI system's attention mechanism called the attention head, which helps the model weigh different pieces of information when generating a response. 

"We derived a 'tipping point formula' for when the crack opens up, and hence the AI output flips to undesirable," Huo said in a media release. "The formula tells you whether an AI is about to flip immediately or whether it will first feed you a run of acceptable answers — and then turn." 

The formula calculates the distance between the AI's current attention direction and the direction associated with undesirable output, and the speed at which its attention moves towards or away from that direction. Dividing the distance by the speed gives an estimate of how much time remains before the system reaches the tipping point. In the paper, the researchers have called the tipping-point formula Equation 2. 

They tested their predictions on seven publicly available AI models, ranging in size from 124 million to 12 billion parameters — numerical values the models adjust during training to help them recognise patterns and generate answers. More parameters generally give a model greater capacity to represent complex patterns, although size alone does not determine performance. 

The researchers report that the formula identified the correct outcome in 18 of 19 tested cases in which the AI shifted from acceptable to undesirable responses. This result, however, comes from a relatively small number of cases and will need to be tested across a wider range of models and prompts.

Studies have documented instances of AI chatbots providing misleading information, ranging from inaccurate medical advice to dangerous guidance on substance abuse and self-harm.

The researchers envisage the formula as a real-time warning mechanism. Its potential value lies in monitoring a system while it is generating an answer, rather than relying on someone to recognise a problem after the response. 

"All of the output from an AI system happens faster than a human's response time," says Johnson. "In a document of 30 pages, the tipping may happen on page 20 to content that is factually correct but harmful. For many future AI use cases, such as AI during surgery, during a military operation, or managing stock market investments, there is no second chance. Noticing after the tipping is useless." 

Early-warning system

Johnson believes the formula could provide the basis for an early-warning system built into an AI model, which could be configured to shut down the system before undesirable output is generated. "The solution is therefore: add a piece of computer code — a code that calculates equation 2 — within the AI so that it calculates equation 2 while it is thinking of its next output," he says. "It is like a motion detector warning light for a driver while they're driving. Current safety approaches cannot do this. Instead, they are like adding padding to the car in the hope that if the car gets near the obstacle, there will be a soft impact."

Studies have documented instances of AI chatbots providing harmful guidance on eating disorders and ways to hide food habits from family.

Such a system could be particularly useful as more people use AI models operating directly on their phones or other devices, without an internet connection. These models may have limited access to safety checks provided by centralised systems. Doctors handling confidential patient information, lawyers protecting privileged information, and soldiers operating without a reliable signal are among those who could have particular reasons to use such systems. 

AI training techniques may change the conditions under which tipping occurs. Such training may suppress tipping for particular combinations of prompts and undesirable outputs. However, a new subject or a carefully constructed prompt could potentially restore the conditions that lead to a change in behaviour. The researchers therefore see scope for using their formula to improve AI training and influence the conditions under which systems generate their responses. 

The findings are encouraging, but come with limitations. The formula correctly predicted the tipping outcome in 18 of 19 tested cases across seven publicly available models. The researchers also examined commercial chatbots, but could not test the formula directly on those systems because their internal workings are not accessible. Further tests will be needed to establish how well the approach applies across a wider range of models and real-world uses.

LEAVE A COMMENT

Search by Keywords, Topic or Author

© 2026 IIT MADRAS - All rights reserved

Powered by ADK RAGE