Frugal AI - examining the pros and cons and how to get it right

The Frugal AI approach is likely to take off over the next 18 months as concern over traditional AI’s significant environmental footprint and high cost profile continues to mount.

This is the view of Fred Lherault, Field CTO EMEA/Emerging Markets, at data storage vendor Pure Storage, which started developing its products using Frugal AI principles about a year ago. He believes that although it is still “early days”, the approach will “be adopted pretty much by everyone” as they move from the proof-of-concept stage to deploying AI “for real”:

One of the reasons it matters to business is because we live in a world where resources are limited. So, even if you’re willing to spend the money, you don’t always get the chance to access all the compute resources you want. There are energy limitations, especially in data centers that aren’t geared to support AI’s power requirements, so anything people can do to limit energy usage helps. Most businesses on an AI journey are stuck in the ‘data readiness’ phase. This means finding it and cleaning it to put it in a state where it can be used by AI. As a result, anything that means you use less data, which includes models built using Frugal AI approaches, will help.

Arjuna Sathiaseelan, Chief Technology Officer at Cambridge Frugal AI Hub, also points to three additional drivers for adopting such principles:

The first is GPUs and spiralling cloud infrastructure costs, especially for sectors operating on thin margins. The second is regulatory, customer and employee pressure. People are now asking harder questions about companies’ carbon and water footprints, and it’s become a compliance and reputational issue. The third is a growing recognition that bigger is not always better for undertaking real world tasks in either robustness or cost terms.

Core Frugal AI principles

On top of the work being undertaken by the Hub to design more resource-efficient, sustainable, and accessible AI systems, Lherault also points to Spec 2314, a methodology and best practice framework for measuring and reducing AI’s environmental impact that was published in June last year.

It was jointly-developed by the AFNOR French standards body in collaboration with 140 organizations. These included NGOs, public sector bodies and private sector companies, such as Schneider Electric (SE). The energy management system supplier now follows three core principles to manage the energy consumption of its AI implementations:

1. Right-size every model: Select the smallest model possible to your meet performance requirements. While it may not have all the capabilities of a larger model, it is important to look at things in the round
2. Deploy workloads at lower carbon sites: Grid efficiency varies widely around the world so running workloads in locations with greener grids using renewable energy sources can have a significant impact on your carbon footprint
3. Build safety that drives efficiency: For example, including safeguards in chatbot interfaces to prevent adversarial use, such as spreading misinformation, not only ensures that systems are protected. It also ensures that carbon footprints are not negatively affected by increasing the overall computational workload of data centers, both due to the attacks themselves and the action required to counter them.

Jeff Willert, Director of Data Science for SE Advisory Services, explains:

It’s about not using a sledgehammer to hit a nail. It’s about rightsizing solutions, so if I can solve a problem without AI or a large language model, I will. I just want to solve the problem, and the solution doesn’t have to be sophisticated if something simple will do.

Lherault agrees but also points out:

If you’re using a smaller model, it’ll be faster and more efficient, but it may not be as accurate because you’re using less data, so it’s a balance you have to find. But it’s also worth bearing in mind that with larger models, you get diminishing returns. This means that just because you use three times more resources, the results may not be three times better. It’s true with data too – sometimes you need 10 times more data to get just incremental benefits. So, if you have a smaller, carefully curated and selected data set, you can still get good results.

Embedding sustainability into everything you do

But Willert is also careful to note that when designing systems, it is important to build in sustainability from the outset rather than add it in later as a “post-development artefact”, which is less effective:

Assess what the impact is as you choose what tools to use and what approach to take, in the same way you would when thinking about conforming to the European Union’s AI Act or Sustainable AI principles. So, think about your model, what type of data you’re processing and what you’re asking the system to do in terms of inputs and outputs as there are different ways to solve problems. Get creative and use a whiteboard as this is an iterative process.

Another consideration is that it is vital to look at each proposed system or product’s carbon footprint in the round. As Steve Wilhite, Executive Vice President of SE Advisory Services, says:

We deployed agents to handle energy invoices for a customer that had been traditionally scraped and captured in their systems. About 5% of the invoices showed anomalies in meter readings that required human intervention. When we deployed ‘Anna the Anomaly Agent’ a year ago, it was initially 56%-57% accurate. Within a month, this had increased to 90% and then higher, while cutting a 10-to-15-day processing timeframe to minutes and seconds. If you think about the carbon content of those 10-15 days of work, analysis and CPU usage, it made sense to move to agentic AI due to the value the client gained and the reduction in its carbon footprint. So, you have to look at these things in the round.

Moreover, he points out that to adopt a Frugal AI approach requires a company culture that has “sustainability built into its DNA”. This means ensuring sustainability is embedded into all business processes as the default:

If it’s not, it runs the risk of becoming window-dressing. It’s important to make it real and down to the bone.

Willert agrees:

Frugal AI will only be successful if it’s completely aligned with your business strategy. AI, sustainability, and your business strategy all have to be aligned. To transition to Frugal AI, take stock of how you’re using, or intend to use, AI and what value it will bring to your processes. Consider whether carbon usage will match the value you receive as carbon and cost are often tightly coupled. How much are you paying, and what is the current carbon footprint of your model? Do I really need AI or am I thinking of using it due to its marketing value? Be honest about it and evaluate things on a case-by-case basis.

My take

While it is still very early days in terms of Frugal AI adoption, the approach could, in theory, revolutionize traditional AI, providing a path to sustainability. But only, it seems, if Sustainability sits at the heart of companies’ strategy – just tacking it on the side and hoping for the best really won’t cut it.