Teaching AI to reason

Subbarao Kambhampati, Professor of Computer Science at Arizona State University, continued to be bothered about large language models (LLMs) after they made a breakthrough in the 2020s. Over the past three years, researchers had noted how a language model could be impressive in answering Olympiad-level questions but extremely stupid in answering basic arithmetic problems. If students could do integration, you would assume they could also do multiplication and division. This expectation of capabilities was not true for LLMs. In 2024, Canadian computer scientist Andrej Karpathy coined a term for this: LLMs, he said, had "jagged intelligence".

"The boundaries of what language models know and don't know are very jagged," says Kambhampati. He understood this was because language models were broad and shallow while classical artificial intelligence (AI) was deep and narrow. "The early systems like Deep Blue or AlphaGo were designed to do just one task but do it extremely well," he says. On the other hand, language models were trained on the scale of the entire internet to perform next-token predictions, such as the next words in a sentence, which would be coherent but not necessarily correct. The lack of an internal world model meant they consistently performed poorly in reasoning and planning. Kambhampati was determined to crack this problem.

In February 2024, he developed an LLM-modulo framework that would add external verifiers to LLMs. Whenever the LLM generated a solution to a question, it would give out a bunch of possible answers. These answers could then be checked by an expert verifier for various classes of problems such as planning, arithmetic, coding, and so on. "If the answer is wrong," he says, "the verifier tells the LLM that this is wrong and here is why. So get another answer." The framework's aim was to provide some sort of guarantee about the correctness of solutions provided by LLMs, especially in the case of reasoning.

This technique is now popular as inference-time scaling. Depending on the complexity of the problem, he was letting the language model increase the time taken to compute an answer. His work turned out to be prescient. Just a few months after its publication, AI research organisation OpenAI released its o1 reasoning models, using inference-time scaling. At its heels was DeepSeek, an AI start-up established in 2023 in Hangzhou, China. In January 2025, DeepSeek released its R1 model, built primarily to incentivise LLMs to reason better. Meanwhile, for much of 2024, Google DeepMind, too, was working on improving the critical thinking of its models, with its AlphaProof and AlphaGeometry 2 coming in second at the International Mathematical Olympiad 2024.

DeepSeek's launch shook rivals such as OpenAI by showing that a reasoning model could be built at much lower costs. Moreover, it was open source, prompting smaller companies around the world to dip their feet into the pool. In India, tech start-up TWO AI, founded by Pranav Mistry, former President of Samsung Technology & Advanced Research Labs, introduced its own foundational model, SUTRA-R0, to make reasoning in Indian language models. Moreover, Paras Chopra, founder of the software company Wingify, recently announced that he was open to collaborations with developers to build an indigenous reasoning model from India.

According to Pushpak Bhattacharyya, Professor of Computer Science and Engineering at the Indian Institute of Technology (IIT) Bombay, the shift towards reasoning is grounded in intuition. Bhattacharyya, who has been studying AI for over two decades, points to a phenomenon known in academia as 'the AI pendulum swinging back'. "Historically speaking, every 20 years, we see a paradigm shift in the way AI is done," he says. The aim of AI was always to solve problems that human beings were good at: from puzzle-solving to music and mathematics. Early AI models focused on searching for solutions by understanding the world they were set in. In the 2000s, when an enormous amount of data suddenly became available on the internet, AI became driven by machine learning. But now, he says, it is time for another paradigm shift. For much of the industry today, it seems that reasoning and planning are the next challenges to be conquered on the way to the holy grail of reliably autonomous AI.

Large language models burst into the AI scene with the 2017 paper 'Attention is All You Need' (bit.ly/google-attention), written by Google researchers, detailing the transformer architecture. The transformer architecture had the ability to pay attention. Given a piece of text, it would discover the relationship between words, however far apart, and so figure out what it should generate as the next word.

In classical AI, the problem is posed in natural language. To perform reasoning, this query needs to be converted to some sort of a formal representation that a computer can understand and then converted back to natural language. The LLMs bypass the middle steps and solve the problems in natural language. "We don't actually know how we think. We just know that finally, when we communicate, we write it in a language," says Kambhampati. The ability of a being to communicate its thoughts to another person is one of the first assumptions of intelligence. So, with more and more training, as LLMs began speaking more like humans, it became easier to assume they were intelligent.

Humans learn about the world not by reading millions of documents but by going out and experiencing it for themselves. While humans learn a lot from just one or two experiences, language models need a large amount of data —pretty much the entire internet today. "Neural networks typically start with a blank slate," says Peter Clark, Senior Research Director of the Allen Institute for Artificial Intelligence, Seattle. "But neuroscientists will tell you that we have a lot of specialist circuits in our brain that are genetically built-in," he says. The institute was established in 2014 by Microsoft cofounder Paul Allen.

Kambhampati believes that the constant anthropomorphisation of machines creates wrong ideas about how they work. "Even using terms like chain of thought in LLMs for intermediary tokens makes people think of the wrong metaphors. These don't have to look like thoughts at all," he says. People trust in the reasoning of LLMs because they speak in phrases that are associated with thinking. "This is a cognitive flaw in humans, and training LLM systems to exploit this flaw makes no sense to me," he says.

For example, LLM's intelligence (or the lack of it) seems to be data-dependent. Since these models train on a corpus of all the information humans put up on the web, they may be retrieving answers to similar questions they have already seen. Kambhampati compares this type of learning to students in India who spend years in coaching centres, where they get trained on all possible varieties of questions asked in the national entrance exams. So, when somebody answers a question in the exam paper, it is difficult to infer whether they have reasoned it through or whether they just happened to have seen a similar question in one of their many coaching lessons. "The only difference is that these students are human, so they'll do something other than just attend classes and may not be able to remember and reproduce everything. LLMs don't have that problem," he says.

Humans are good at learning by observation — like a person who moves from one country to the other and learns to switch from the left side of the road to the right after watching the position of the steering wheel in a car. Common sense dictates that, in this new scenario, the fastest lane would be on the left side, and new drivers learn to overtake other vehicles from that lane. This is because people essentially understand the concept of driving and the rules of traffic. This kind of common sense is nearly impossible to see in LLMs without any data points of what left-hand driving is.

"Causality is faked in a lot of these models," says Balaraman Ravindran, Professor at IIT Madras and head of the Robert Bosch Centre for Data Science and Artificial Intelligence. That's because a causally valid sentence can be generated just by going with the documents the machine has seen already. "The universe of documents is typically causally valid," he says. "So, if I'm just going to use those probabilities to generate answers, their causality will anyway be maintained."

Causal reasoning becomes an emergent property from the way the models have been trained, even if they were not doing true reasoning in the beginning. "Humans themselves are bad at doing causal inference," argues Ravindran, adding that people tend to go with correlational input and then do some sort of backwards reasoning to prove themselves right. "We think that we go from the rationale to reason," says Ravindran, "while more often than not, we first make the decision and then go back and find a rationale for it." Humans judge people by their appearance. However, when asked to explain why, they come up with rationales such as a seeming lack of education. "The question is: if AI is going to do the same thing, are you happy with it, or do you want to hold it to a higher bar?"

Kambhampati's models reward correct reasoning, a reinforcement learning technique used extensively by companies such as DeepSeek. Whereas he proposed the addition of external verifiers during inference in AI models, DeepSeek and others are now using verifiers to reinforce learning during the training stage itself. Here, verifiers decide whether or not a reasoning path ending in a solution ended up producing the correct solution. If it is the correct solution, the model rewards that reasoning path. If it is the wrong solution, that path is penalised. Reinforcement learning is how language models were made to understand how to avoid the racist, sexist and dangerous content they picked up from the internet. A similar concept is being applied now to reasoning.

The need for causality in AI is especially important when employing it for scientific and mathematical discoveries, where deeper deliberation is required. Researchers can now pose a question on, for example, the correlation between altitude and temperature, and it would give a correct answer. "But that is not because these models understand the physics to give you that answer," says Amit Sharma, Principal Researcher at Microsoft Research India. "They have just become very good at retrieving causal facts that might have been in their training data." Current AI models, on the other hand, do well for tasks that need creativity and require a wide — even if shallow — base of knowledge. However, they may not always be sound or accurate. "Creativity and factuality are often at loggerheads," says Kambhampati.

Sharma describes this as the soundness vs completeness problem. For any intelligent system, there are two properties that need to be satisfied. One is soundness, which means that the system gives a correct answer. The other is completeness, which means that the system gives an answer – no matter what the subject is. Throughout AI history, people have built systems that were sound first and then tried to increase completeness. "You'll say. 'I'm great at one task, now I will move to the other'," says Sharma. "LLMs go the other way around. They are close to being complete, but they are not sound. And I think that is what is making it really hard for us to evaluate them or to even understand what kind of reasoning they are doing." To bring causality into LLMs, Sharma has been tinkering with introducing logical training to transformer architectures. This involves teaching basic rules of logic to an LLM.

For AI models to move beyond acting as search tools for research, and creating new knowledge, they need to be able to look at the existing knowledge and then combine two pieces of knowledge in a way not combined before. "If we teach some rules of logic to the model," says Sharma, "they will know what to combine in a way that has a much higher probability of giving interesting results." In this technique, Sharma creates synthetic worlds in the form of causal graphs and axioms. The graphs help generate training data in natural language that describe the worlds and the properties they should obey. A language model can then be trained on this data to learn about these worlds. For instance, a language model can be trained that if A implies B, and B implies C, then A should also imply C. And also that C need not imply A.

Sharma found that when they trained language models on data where the graph size was only up to six nodes that were related to each other in some causal way, the model could understand the chain of reasoning up to eight nodes without previous training. "It is very early work, but this is the sort of direction I'm very optimistic about," says Sharma. "Rather than training LLMs for every new task, we train them on some very fundamental rules or basic theorems, and hopefully, they will learn to apply these theorems recursively or apply them multiple times."

The final system is still a transformer but one that, at training time, uses data generated from a symbolic system and converts it to natural language. At the Allen Institute, Clark has tried something similar: adding a layer of rationality on top of LLMs. In 2023, his team published a modified LLM architecture, called REFLEX, which builds a reasoning layer on top of the foundational layer. In essence, this model tries to embody the two kinds of human thinking popularised by psychologist Daniel Kahneman in his book Thinking, Fast and Slow. The base LLM layer would be fast and instinctive, while the top layer would be a deeper and deliberative one.

Their work started in 2019, after Clark and his team built an LLM that could solve a Grade VIII science exam — a big feat for the time. However, worried that the model was not able to understand the question and was possibly faking its way to a good grade, the team started probing what it was really doing. One way to do it was to ask several questions until they had a network of all the things the AI believed in and how they were related to each other. Doing this at scale produced a big graph that summarised everything the language model knew. "That's how we found out that language models can be inconsistent in their beliefs," Clark says.

The team added a reasoning component that would sniff out contradictions in the inference and work it out by changing some of the problem-causing beliefs. "We ended up with this two-layer system where we have a language model doing all these instinctive things at the bottom, and then you have a rational layer on the top which summarises the language model's beliefs, reflects on them, and tries to fix them," Clark says.

The most popular ideas to improve the reasoning capabilities of AI still revolve around tweaking the transformer architecture. "In technology, there is a system of lotteries. LLMs have won the lottery for now. And more people are likely to work on things that are already working, even if better solutions are possible," says Kambhampati, comparing it to the popularity of airships until the Hindenburg fire.

In 1937, a German airship crashed and burned in New Jersey, killing 36 people. It ended the era of large airships for commercial travel, paving the way for the modern aeroplanes in use today. This is happening in LLMs, too, to some extent. "If, at some point, LLMs fail on such a big level, that is when the other smaller models can come into play," Kambhampati says. At a smaller level, DeepSeek did try to bring in change using a mix-of-experts architecture, where the model has specialised sub-networks that independently perform different parts of the task at hand, much like how the human brain works. So, the change to completely newer architectures might happen over a period of time.

A small number of researchers are working on out-of-the-box solutions. Yann LeCun, inventor of convolutional neural networks and Chief AI scientist at Meta, has been advocating for students to move away from LLM research towards world-building. In 2022, he proposed the Joint Embedding Predictive Architecture (JEPA) to build a hierarchical AI system. LeCun envisions a system that uses neural networks to interpret data, builds a world model according to it, and uses that model to plan its next sequence of actions, in a hierarchical manner. The idea of building world models has been popular with DeepMind as well — the company introduced the MuZero algorithm as a kind of advanced reinforcement learning algorithm that could learn its environment's rules on the go, and make decisions accordingly.

At the MIT-IBM Watson AI Lab, the focus is on neuro-symbolic AI. Researchers from IBM developed Logical Neural Networks (LNNs) to perform logical reasoning. Although LNNs were introduced in 2020, their roots are much older: the idea of neuro-symbolic AI took hold at the same time that neural networks did. Neuro-symbolic AI combined the classic symbolic AI with modern neural networks.

"These are two complementary approaches to learning," says Dinesh Garg, who manages a research group on Neuro-Symbolic AI & Reasoning at the IBM Research Lab in Bengaluru, India. For symbolic AI, the goal is to search. It is fed the rules of the game and looks for the right solution to achieve its objective. This is how the classic chess-playing machines were built. In the neural network school of thought, however, there are no rules: it is all based on learning the pattern from the data and employing it to perform inferencing.

"Take, for example, an image classifying network. You don't feed the system rules on what the shape of a cat or a dog is. You simply give it the images and the label, and let it figure it out on its own," says Garg. With neuro-symbolic AI, the objective was to somehow combine these two approaches and build a machine that could handle both kinds of challenges at the same time.

While Garg started by working on game theory and symbolic AI, as machine learning started picking up, he became interested in the challenge of getting AI to solve complex optimisation problems using neuro-symbolic techniques. In 2021, he, along with other researchers, built 'Explanations for CommonsenseQA' (bit.ly/commonsense-ai), a dataset to teach AI systems how to reason about the answers to everyday commonsensical questions. "The idea is to build a machine that can think like the human brain," he says — a lofty and, so far, formidable goal.

Such reasoning models become more reliable when the required tasks are grounded in the physical world as opposed to cyberspace. Planning is important in worlds where there are high costs associated with failure, as there are in the area of robotics: the physical world generally tends to be less forgiving than the cyber world. Garg points to self-driving cars, which need to be "thinking fast" by looking at the road through their sensors and accordingly operate the brakes or accelerators. "But at the same time, they should also have the capability to deal with contingencies. If it encounters something unknown, it should chart out an alternative path using slow deliberation," says Garg.

Perhaps the most attractive feature of reasoning models is the biggest constraint of transformer architectures: explainability. To regulate AI, you need to know what's under the hood, and for that, you need to have models that can explain how they work. Without this, building trust in AI systems remains a challenge.

"When we ask someone a question, and they reply with a step-by-step account of how they arrived at the answer, it is easier to trust them. The same stands true for AI," says Bhattacharyya of IIT Bombay. Moreover, instead of going to a direct final answer, such systems give you the intermediate states of computation, where you can discard what is not fruitful. "These are the big draws of reasoning models: efficiency and trustworthiness," he adds.

This is especially true in cases where critical decisions are made, such as healthcare, law, and finance. "Wherever wealth or health is involved, people want explanations for the decisions that will affect them. Without causal reasoning, these three sectors will not get much currency from AI systems," says Bhattacharyya.

In medical diagnostics, AI models are used to look at tissue biopsies and discern which ones are cancerous. In such life-and-death questions, patients would want to know the basis of its verdict. A doctor shown the image may look at other contextual factors — rules acquired through education — before making a decision. "With models that can reason, and spit out the explanations behind their decisions, an expert human has a second chance to course-correct in case of mistakes," says Garg, "And that can make all the difference in the world to the affected person."

See also:

AI for a reason
AI models for India
The open-source edge in AI
Teaching physics to AI
All's not fair with AI