RAG to riches - coding a solution to AI’s privacy problem

One week after DeepSeek released its eponymous AI chatbot in January, it became the most downloaded free app on the iOS App Store in America. That same day, I announced publicly that I would be changing roles — from CEO to Chief Scientist — at Zoho Corporation, the technology company I founded and led for nearly 30 years. DeepSeek's sudden impact could be felt across our entire industry, understood by many tech leaders as a profound disruption. As an engineer first, I viewed DeepSeek's perceived disruption as a pathway toward innovation.

What DeepSeek achieved with its R1 model is a milestone of efficiency, not of capability. DeepSeek's model isn't objectively better or worse than OpenAI's, but it cost $6 million to build, which is $94 million less than ChatGPT. More than just being cost efficient, the R1 model runs on 90% less compute power than Meta's Llama 3. Together, DeepSeek's efficiency leaps have profound implications for AI's future and potential in both the consumer and business contexts. It also means that everybody's 2024 predictions about AI were wrong.

Chief among last year's predictions-cum-hallucinations was a notion that Big Tech, and only Big Tech, would have the capital to own and control the AI market. That market's size globally sits around $250 billion, $80 billion of which Microsoft alone will invest in 2025. The assumption from onlookers and the intention of providers was that customers would foot the bill for these enormous Big Tech expenditures, paying ad-hoc for AI apps and functions. Flash forward to this year, AI features that use reasoning to bring intelligence to a problem have become table stakes. Where is the value? How much can Big Tech charge today for services DeepSeek and others can provide for a fraction of the cost and power? I believe the window to charge anything or market AI as a differentiator in good faith will be closed by the summer.

This fact that large vendors can no longer monopolize and monetize reasoning intelligence has shifted the competitive battlefield from controlling AI access to delivering superior data-driven, intelligent assistance. This is why we're seeing such interest in agentic AI, even though intelligent assistants have been around for a decade. Agentic AI is a powerful and potentially transformative use case for the technology, particularly in a business context. It's not hard to imagine a single organization utilizing thousands of task-specific agents across departments or for particular employee workflows: AI sales agents gathering, categorizing, and developing leads; agents to transcribe client meetings, automated to send summary emails, schedule follow-up meetings, source materials, prepare documents, etc.

Privacy in peril

Sounds great, right? At this year's South-by-Southwest tech industry and enterprise conference, Meredith Whittaker, president of the private messaging provider Signal, laid out a foundational problem impacting agentic AI very well. "I think there's a real danger that we're facing," he said. He continued:

The value add is something like, it can look up a concert, book a ticket, schedule it in your calendar, and message all your friends that it's booked.

To do this, Whittaker explained, it would need access to the user's browser and credit card information, each invitee's personal calendar, and finally, the user's messaging platform:

It would need to be able to drive that across [an] entire system with something that looks like root permission, accessing every single one of those databases, probably in the clear because there's no model to do that encrypted.

Whittaker went on:

[It's] almost certainly being sent to a cloud server where it's being processed and sent back. So there's a profound issue with security and privacy...

At the conference, Whittaker was sounding the alarm about agentic AI for consumers. In a business context, organizations are liable for the protection of customer data and the security of their own. They can't, in other words, ensure any measure of protection when private data ping-pongs from one third-party app or service to another, out of view and impossible to track, in service of task productivity. Issues of privacy and security currently obscure the full picture and potential of sophisticated, highly capable task automation and reasoning intelligence in business workflows.

For one, how do you delete information from an LLM? Even if an LLM is personal to one organization, or even to one user, there's currently no way to configure that model to forget, much less set access permissions to the data the model has been trained on, case-by-case. The more powerful the LLM, the less transparent it becomes and the harder it is to regulate. LLMs show incredible promise for industries like government and healthcare, but ensuring their use is HIPAA, FedRAMP, or GDPR compliant becomes an insurmountable obstacle.

A two-brained system

So here we are in 2025. DeepSeek has proven powerful and efficient models can be affordably developed and deployed; public LLMs like ChatGPT have raised awareness and generated enthusiasm from businesses and consumers; and potentially transformative use-cases for agentic AI abound. How then do technologists approach the 10,000-pound privacy and security elephant that's haunting AI's future and halting its potential business value?

Since moving to Chief Scientist, I'm attacking this puzzle first by utilizing a two-brained system, so to speak. The thinking is that one siloed and secure database provides input data, and an entirely separate AI engine and LLM outputs reasoning and generated influence. To accomplish this AI brain surgery, the LLM driving output must be a black box — one pre-trained with general purpose information, but never exposed to customer, personal, or proprietary data. Only after a user or AI agent prompts that system to generate content or prescribe an action will the black box source data from the input LLM, employing a Retrieval Augmented Generation (RAG) framework to deliver precise, contextual results from data it cannot store.

This two-party approach is not new, but it does ensure a layer of privacy and security that is fundamental for business use. This approach does not, however, guarantee that the output is accurate or verifiable, just that it's private. Generated RFIs or legal briefs, for example, would still require copy editing from a human being with knowledge of the content's context. The moment a person is needed to verify an engine's output, productivity grinds to a halt. It's an extra step that only gets easier the better LLMs become, but it never goes away.

The new AI space race

Why agentic AI has gained so much attention recently, at least from engineers, is that agents don't prompt LLMs for generated written or visual output. Instead, AI agents engage LLMs for intelligent, pre-scripted actions or contextual next steps — book a flight, file and expense, send a follow-up email, etc — which the model provides in the form of code, or structured data. Why does that matter? Structured data is binary; it can be machine-verified, AB-tested, and trained using RAG architecture to autonomously guide an AI agent to perform appropriate actions in any context, every time. Engineering this level of consistent, useful, and secure agentic autonomy that serves business end-users is a mammoth undertaking and remains to be seen.

This, to me, is the new AI space race. Whoever can develop a machine-verifiable, private and secure LLM with an intelligent assistant first, wins. Given the immense advantages and extreme productivity programmers have gained with AI already, this race is going to be a sprint. So with DeepSeek as proof of how quickly things can change, the field is open and an opportunity is available for anyone whose objective is innovation.