More for less

With companies embedding artificial intelligence (AI) into multiple applications, from customer support and coding assistants to basic search and productivity tools, overall AI budgets are exploding. For every prompt entered, AI agents consume computing power, measured in tokens. Ride-hailing technology company Uber famously burned through its AI budget for 2026 within four months on coding assistants, and was forced to cap AI usage per employee. "Cost per token has become the biggest mantra for companies using AI in their workflows today," says Kamalakar Devaki, Founder of SandLogic, a Bengaluru-based AI enterprise company.

"There have been reports of companies spending huge sums because agents went into rejection loops or made errors. We have also seen cases where AI agents generate significant costs without delivering the expected value. All of this points toward one thing: token economics is becoming a serious issue," says Devaki.

While India is creating its indigenous AI models and applications (see story on AI for India), the inference, or deploying these trained models, still mainly happens on foreign hardware, raising costs along with it. Of the ₹10,372 crore committed by the Central government under the IndiaAI Mission, nearly 44% is directed towards subsidising domestic start-ups and academia's access to high-end computing power.

"We cannot have a situation where only a handful of large companies can afford world-class compute. Start-ups, researchers, universities and public institutions should all have access to the same capabilities so innovation can come from anywhere," says Sunil Gupta, Co-founder and CEO of Yotta Data Services, an AI cloud infrastructure company. Private players like Yotta have partnered with tech giant NVIDIA to provide graphics processing unit (GPU) clusters through their Shakti Cloud platform. These clusters are supplied to organisations that train models, such as Sarvam AI and BHASHINI. Gupta shares how Bhashini, which had been using foreign hyperscalers until recently, migrated 200 TiB of its language datasets and models to the Shakti Cloud AI infrastructure, saving about 20-30% in costs.

The company also introduced Shakti Studio – or what it calls 'an AI Token Factory'. Instead of a flat computing fee, Shakti Studio bills its clients for tokens used, or GPUs used, per minute. "The idea is to simplify AI adoption for enterprises. Not every organisation wants to build foundation models from scratch. Many want to quickly create AI agents, copilots and agentic workflows that solve specific business problems," says Gupta.

Expanding and subsidising access to compute is only part of the solution. On the other hand, companies are reducing costs associated with the hardware through engineering. At SandLogic, Devaki believes that to truly have a say in how much AI costs your company, you need to have control over every layer of the AI stack.

As a vertical-stack company, SandLogic is building its own AI chips, its compiler, six AI models, and the inference engine to run them, and is helping enterprises with AI applications. "When a user submits a request, every layer has a responsibility — from the hardware to the inference engine, the model, and finally the application," says Devaki. Improvements at each layer affect the next, he explains. "If you don't understand those gaps and solve them, you fall into the same trap everyone else does: using the same hardware, training the same way, and accepting whatever costs are imposed on you. That cycle needs to be broken."

The inference engine, for instance, acts as a bridge between the hardware and the model. It has to understand what the model requires and what the hardware can provide. But inference engines can do much more, says Devaki. They can be designed to remember information across multiple requests without having to go through the whole conversation again. This reduces the number of tokens used – and the inference cost.

Most of SandLogic's customers use its applications layer alone as it has not yet taped out its system-on-chip. But companies that want their full stack are those building solutions that will come to the market two to three years from now. An automobile manufacturer it is working with has asked the company to design an AI voice agent to be run locally in the car, rather than offloading the processing to the cloud. This way, it can work in places with no internet connection.

The company's foray into AI hardware dates back to the Swadeshi Microprocessor Challenge, launched by the Central government in 2020 to spur the technology-led innovation ecosystem. It was a few years after a team at the Indian Institute of Technology Madras, led by its current Director V. Kamakoti, developed the open-source Shakti microprocessor based on the RISC-V architecture. The family of processors was designed as an alternative to proprietary architectures licensed by ARM, AMD and Intel. The microprocessor challenge incentivised companies such as SandLogic to build hardware using Shakti and C-DAC's Vega processors. SandLogic added AI capabilities to Shakti to build their first version of an AI accelerator for object recognition and classification.

The Shakti architecture gave rise to many others with similar ambitions of building indigenous chips. One of them is Chennai-based Mindgrove Technologies, co-founded by Shashwath T.R. and Sharan Jagathrakshakan. The duo is building edge inference processors designed for AI workloads. By using open-source processor architectures, these companies hope to reduce AI inference costs.

The AI hardware coming from India, however, is geared more towards edge use cases and smaller models. "Models are multiplying like rabbits, while chips are evolving at the pace of elephants," says Devaki. This huge gap between the two means that chip builders need architectures and design systems that can accommodate future models. "India is a resource-constrained market. Larger economies may be able to absorb inefficiencies, but India cannot," he adds.

Another way is to rethink processor architectures beyond the traditional central processing units (CPUs) and GPUs. Morphing Machines, founded at the Indian Institute of Science, Bengaluru, by Ranjani Narayan, S.K. Nandy and Deepak Shapeti, has developed a specialised processor, REDEFINE. Called XPUs, they perform dynamic computations depending on the application. The goal is to reduce power inefficiencies and latency.

At Ziroh Labs, Co-Founder Hrishikesh Dewan questions the assumption that companies need expensive processors to run. With Kompact AI, Dewan has found a way to build AI on CPUs, rather than GPUs, at lower cost. "For the last four or five years, we have been trying to find mechanisms where you are not required to compute so many numbers to generate the same token," says Dewan.

Kompact AI is not trying to replace GPUs, he emphasises. For workloads like training models, or applications that need to support over 10,000 users, there is no way to get around it. But it would be useful when a company is building a solution that needs to be installed on-premises or in the cloud, where fewer than 100 users will use it simultaneously, and for models of no more than 32 billion parameters.

"In these cases, if you are using GPU clusters, you are overpaying for the application you are deploying," says Dewan. "It's like buying a superbike for ₹40 lakh that can go up to 300 km/h and then driving it on suburban streets at 10 or 20 km/h." It's a reminder that cheaper AI does not necessarily come with easier access to compute, but rather from knowing when not to use it.

See also:

Made-for-India AI

AI for India

No country is fully sovereign in AI: Maya Sherman