Several ways to predict when AI chatbots will go bad

Researchers at George Washington University propose a formula for the "deterioration" critical point of AI dialogue models.

Physicists at George Washington University have published a new formula to estimate how many correct answers an artificial intelligence chat robot can provide in a row before giving a wrong answer. Early testing showed that the formula was effective to some extent.

The study, co-completed by Neil Johnson and Frank Yingjie Huo, was published in the journal Patterns and was based on a preprint version previously released in February. Research has shown that chatbots can answer questions reasonably for a long time, but may then turn to harmful content, such as bad advice about self-harm or extremist remarks. Currently, the industry lacks a simple way to predict when such a turning point will occur.

The authors point out that existing security tools often rely on cloud connectivity, while models running offline do not. Johnson and Hu attributed the root cause of the problem to the model's "attention head". This is the part of the AI model that determines which previous conversations are most important when generating the next word. As the conversation progresses, the cumulative context draws attention to a certain type of answer cluster until a "tilt" or mutation occurs.

This is a common pattern exploited by many attackers trying to circumvent security restrictions, and one reason why many technology companies attach great importance to the system prompt word (a piece of text that is entered into the AI before a user queries). However, no one has yet been able to accurately point out how much effort will be needed to effectively weaken the model's security mechanisms.

Critical Point Estimation and Test Validation

Their formula defines a critical point (called n) as the number of good "tokens"(i.e., one word fragment generated by the model at a time) before the first undesirable output occurs. If the conversation has tilted in the wrong direction, the model will tilt immediately, and the value of n is zero. If the conversation is in a good direction, the model outputs a series of good answers first and then suddenly rolls over.

In preprint testing, the formula correctly judged immediate tilt or delayed tilt in 16 clear and clear cases, with an accuracy rate of 94%. The researchers ran the tests on six publicly weighted models. The so-called "public weight" means that the files of the AI system are public and anyone can download and run them. These models come from institutions such as OpenAI, EleutherAI, and Meta.

The parameter quantities of these six models range from 124 million to 410 million. Parameters are numbers that can be adjusted within the model and are used as rough indicators of model size. According to reports, the officially published paper expanded the test to seven models, with a maximum parameter volume of 12 billion, although this is still a smaller-scale model by current standards.

Security monitoring for device-side AI

The focus of this research is on on-device AI, AI that runs entirely on a mobile phone or laptop and does not require an Internet connection, including chatbots that accompany users like friends. Google's experimental AI Edge Gallery app already allows Android phones to run models offline and input content is not sent to Google servers. This trend is likely to grow over time as hardware becomes more powerful and the capabilities of small AI models increase.

Models running offline have no cloud services to check their output, which is exactly the gap the author hopes to fill. They proposed a low-cost monitoring scheme that runs in parallel with the model and issues warnings when the estimated n* falls below a safety threshold, similar to a warning light on a car's dashboard.

In addition, they describe ways to push critical points into inaccessible areas, such as injecting specific content into the conversation so that the value of n* exceeds the length of the response. The authors say that although alignment training (the process of teaching a model to behave) can change or suppress tilt for specific prompt words, it cannot eliminate its underlying mechanism.

Research Limitations and Background

As early as April 2025, the media reported another paper published by the same pair of researchers, pointing out that polite terms such as "please" and "thank you" have little impact on the output of the model, because the model mathematically treats polite words as orthogonal variables that have nothing to do with the request. That paper modeled only a single, deliberately simplified attention head.

此次预印本的测试使用了小型模型和300个标记的窗口（即几段简短的文字），其预测结果可能存在一个标记的输出误差。