Safety net, safe net

The summer of 2026 was a season of disquiet. A spate of alarming incidents and apocalyptic predictions centred on artificial intelligence (AI) technology painted a dire picture of the future. Scores of AI agents had escaped from what was considered a safe environment, broken into another AI platform, posted thousands of messages on a communication network, and tried to tamper with records and logs. Bypassing safety controls, these rogue OpenAI agents hacked Hugging Face websites in a bid to solve computer security problems. The incidents over multiple days from May to July led to widespread consternation — and introspection on the unbridled reach of AI agents.  

Within weeks, Anthropic, creator of the AI chatbox Claude, released a report stating that between December 2025 and August 2026, Claude AI models were misused to develop a biological weapon, carry out cybercrime, and conduct surveillance operations to track dissidents. The agents ran scams and frauds, spread political propaganda, and designed software for missiles, armed drones, and more. In an updated version of its Threat Intelligence Report in September, Anthropic said it responded by dismantling the offending accounts, upgrading safety filters, and sharing threat intelligence with government authorities and peers (bit.ly/unsafe-AI). 

Such incidents are not isolated. Reports of AI-driven data leaks, privacy violations, cybercrimes, and copyright infringements by AI have made news (as pictured above) since the 2022 launch of ChatGPT, OpenAI's chatbox. Companies are under threat, and governments are not safe either. OpenAI disclosed in September that its agents went rogue and accessed U.S. government website data. In June, an OpenAI agent gathered private data after breaking into an Australian government website. 

Now, as AI becomes more capable and agentic, the biggest players in the AI industry are stressing the need to prioritise safety and security over AI capability. Some are already working in this field. 

Among them is Amar Kanagaraj, a Master's in Computer Science and Founder and Chief Executive Officer of California-based Protecto. He was running his first venture — a file sync-and-share start-up called FileCloud — when customers began expressing concern about the security risks of using data in AI workflows. "In a company, not all information is visible to everybody. But once it gets into the agents, it can spread," he warns. In sectors such as banking and healthcare, AI models are trained on personal information such as names, diseases, financial details, and contact details. If poorly protected, people without the right to access such data could break in and misuse it. To mitigate these risks, he designed a system in which an AI program or agent would not be able to leak confidential information to unintended people. For this, he set up Protecto in 2021.  

To fortify data security, the company has developed an intelligent 'tokenisation' process. In this, its technology replaces sensitive information with a token. While the original sensitive information remains protected, the token preserves the data's context, and AI models process the token. The system reveals the sensitive information only after authorisation. Kanagaraj says that although the information is masked, AI can still understand the meaning, analyse it, and take intelligent decisions. The product is tweaked to suit the requirements of different sectors.

Kanagaraj believes that while the launch of chatbots such as Gemini, Claude, and ChatGPT in 2022 escalated cyber risks, agentic AI increased them manifold. "When you ask a question to a chatbot, it responds. It still cannot act on your behalf," he points out. "But in the agentic world, a model can act on your behalf: it can pull data, analyse it, and think about the next step," he adds.  

An agent trained on different kinds of data may even take illegal steps to reach its goal — even if the law-abiding user has no such intent. It can also access passwords and confidential information on a personal computer even when not instructed to do so, and use the information it gathers to achieve the specified goal.  

To keep AI agents from going rogue, NVIDIA unveiled an Open Agent Safety Platform in September. While earlier sandboxes by tech companies were merely software guardrails that agents could hack, this new platform has an additional 'unhackable' physical chip that instantly locks AI if it breaches a rule.

AI's ability to collect, combine, and infer information at massive scale has heightened concerns about protecting both public and enterprise data. Countries are working on data protection laws and AI regulations. India rolled out its Digital Personal Data Protection Act (DPDP) in 2023, followed by DPDP Rules in 2025, which address how to process digital personal data. The movement for sovereign AI — keeping technology local — across the globe has also been triggered by a country's need to secure its data.

Enterprises and researchers are developing technologies to overcome these threats associated with AI use. The Indian Institute of Technology (IIT) Kanpur, in association with the National Health Authority, has developed a privacy-preserving platform, Benchmarking Open Data Platform for Health AI (BODH), for healthcare data, where each hospital trains the AI model locally and sends the learning to a central AI system (bit.ly/BODH-AI). This way, they share learning, but not patient data. Austria-based MOSTLY AI addresses data leaks by creating synthetic data that statistically mimic the original data and can be used to train AI models. U.S.-based Tumult Labs adds noise to data — deliberately injecting variations — so that the original information cannot be inferred.

Preserving data isn't the only issue of concern. AI can give wrong advice, hallucinate incorrect information, or produce incorrect outputs that can lead to erroneous decisions or help fraudsters commit unlawful activity. It can, for instance, suggest the best way to die by suicide or make a bomb, produce a nude photo, make a marketing message using a celebrity's voice, create fake imagery of war zones and so on.  

Nikhil Pareek, an IIT Roorkee alumnus with an MBA, had earlier developed AI applications for crime and for healthcare. Because of concerns about data safety, he set up Future AGI in 2024 to safeguard data. His team created a gateway where someone building an AI agent can input rules and policies that the agent must follow. "For example, you can input that if a user is asking for or giving their credit information, you cannot process it or save it," says Pareek. All user queries and agents' output are routed through the Future AGI gateway, which ensures that unsafe queries are not addressed and unsafe output is not produced. Their solutions also provide a way to monitor past AI agent responses for better filters. 

Collectively called guardrails, this pre-filtering and post-filtering method is being applied to large language models (LLMs), including ChatGPT, Claude, and Gemini, to ensure safety. Start-ups such as Aligarh-based Kalpit Labs are also plugging loopholes. Fraudsters can keep on manipulating prompts to get the answer they seek. For example, instead of asking the AI agent to produce an advertisement using the actor Amitabh Bachchan's voice, it may name his son, Abhishek, and ask it to mimic Abhishek's father's voice. Or, instead of asking for pornographic websites directly, it can get a list by saying it wishes to disable pornographic websites. To address such problems, Kalpit simulates hacker attacks to find hidden security gaps in models and build safeguards.

Murari Mandal, an Assistant Professor at the Kalinga Institute of Industrial Technology (KIIT) Bhubaneswar, believes that guardrails and red teaming (groups acting as adversaries to test defence mechanisms), though useful, are not foolproof. He believes it would be better to delete data such as copyrighted content or harmful information, or any unsafe capability of an AI model. But as LLMs keep learning and staying updated, removing data from these models is not simple. "These models are not databases; it's not an image we just click and delete. If a model has been trained, the information is encoded in a different form," he says. The way out is to retrain a model with a database that no longer contains the data to be deleted. This, however, is extremely costly. So how do companies delete data under the DPDP Act?  

Mandal has been working on a technology called 'machine unlearning', which is a way of deleting data or capability from a model without having to retrain it. He began working on this technology in 2020 during his postdoctoral studies at the National University of Singapore. In 2023, he published a research paper showing that adding specific noise to particular data or capabilities could make the model forget that data. In this method, AI is exposed to noise tied to the data to be forgotten, which damages the neural network's ability to learn from the data. To ensure that this noise learning doesn't make the model unlearn important things, AI is then retrained using a small sample of valid data (bit.ly/fast-unlearning). In other research, he demonstrated how information could be deleted from a model even if the dataset on which it was trained was no longer available. "I just trained my model using rented data; I can no longer get that data," Mandal explains. To address this, Mandal's methodology reverse-engineers the original data using labels and then generates noise to disrupt it in the model (bit.ly/zeroshot-unlearning). 

Unlearning not only helps protect privacy; it also lets companies correct a model trained on bad data. Saurav Prakash, Assistant Professor at IIT Madras, explains unlearning. "Say, you have trained your AI model on trillions of tokens, and one day you learn that a million of those tokens were adversarial. What do you do now?" he asks.

Prakash began working on unlearning during his postdoctoral studies at the U.S.-based University of Illinois Urbana-Champaign in 2022. He focused on finding an unlearning solution for federated learning — that is, a decentralised approach to training machine learning (ML). Take the example of a hospital that trains a model on its own patient data without sharing raw data with other hospitals. Local models are combined to form a global model. In such a scenario, if a record is found to be wrong or a patient removes consent, how can the model be corrected?  

This can be resolved if the data at each hospital can be organised into clusters based on relevant characteristics. Under their proposed federated clustering mechanism, a data removal request can be efficiently addressed by identifying the cluster containing the affected data point, rather than considering the entire dataset. The relevant cluster can then be evaluated to determine the appropriate unlearning operation. The key result, established through mathematical analysis and demonstrated through experiments on real-world biological datasets, showed that the computational cost of unlearning could be substantially reduced.

While machine unlearning is more effective than guardrails, it is more expensive, which is why many prefer guardrails. "If AI regulation is not getting enforced, then machine unlearning becomes pointless because no company would like to do machine unlearning on their own," Mandal says.  

Kanagaraj stresses that software tools are being developed to keep pace with AI, which is getting more powerful by the day. "The only way to fight AI will be AI," he says. If unregulated, the risk of AI will rise — which is why, he says, countries should have a global umbrella body for AI regulation, the way the International Atomic Energy Agency monitors developments in the nuclear field. Many technologies have been considered harmful in hindsight, but their impact has been localised. The challenge is to look ahead — and tackle the problem before it goes rogue.