Databricks takes on the agentic AI challenge - with automation, observability, and scale in mind

Databricks has announced a launch that signals a shift from generative AI experimentation to production-scale deployment – anchored by two new tools, Lakeflow Designer and Agent Bricks. Both are aimed at solving practical bottlenecks in data and AI adoption: the shortage of engineering talent, the growing complexity of AI agent architecture, and the persistent need for trust, observability, and governance.

While the AI market continues to generate heat (and no small amount of hype), Databricks is betting on grounded infrastructure over glossy interfaces. As Richard Shaw, Technical GM and Field Engineering lead for UK and Ireland at Databricks, put it when I spoke to him: 

This isn’t about agent-washing. Our customers are already doing agentic AI – what they need now is a way to do it at scale, with confidence and visibility.

Lakeflow Designer is Databricks’ new low-code/no-code pipeline builder. On the surface, it resembles a familiar category – a drag-and-drop interface designed to help business users ingest, transform, and route data without writing code. But Shaw emphasizes that the real power lies under the interface:

This isn’t a ‘lite’ tool that creates throwaway pipelines. It generates real, observable code that engineers can manage, version, and deploy just like any other pipeline.

Transparency for users

That shared transparency is a core theme. Business users can create pipelines using visual tools, but behind the scenes, Databricks automatically embeds best practices – resilience, self-repairing capabilities, lineage tracking (which shows the history and origin of data), and data quality checks. “You’re bridging the skills gap without building a black box,” says Shaw. He continues: 

Both the business analyst and the data engineer can look at the same pipeline, understand what it’s doing, and fix or extend it without needing to start over.

This democratization of ETL (extract, transform, load processes) isn't just about convenience. It’s a strategic play for scalability. With data engineers in short supply, Lakeflow Designer allows teams to scale their data operations laterally – freeing engineers to focus on complex architecture while empowering analysts and product teams to move independently.

The tool also integrates tightly with Unity Catalog, meaning governance and lineage remain intact. 

If Lakeflow Designer democratizes ETL, Agent Bricks aims to do the same for AI agents. The pitch is simple – building AI agents today is hard, fragmented, and time-consuming. Developed by Mosaic AI Research, Agent Bricks uses novel research techniques to automate the entire lifecycle – from model selection and training to synthetic data generation, performance scoring, and observability. As Shaw elaborates:

It’s like choosing ‘text summarization’ as a use case, and having the system configure everything needed to support that agent at production scale.

Judgement calls

This includes selecting the optimal model, training it with enterprise-specific data, generating synthetic data for better coverage, and setting up observability features such as LLM Judges – models that score other models on output quality.

Those “judges” are especially important. Shaw notes: 

They’re not just for reporting. They give you actionable confidence. You can see if your agent’s performance is declining, figure out why, and adjust accordingly. It’s part of moving from experimentation to sustained production.

Agent Bricks also aims to address a growing challenge in enterprise AI – moving from isolated prototypes to orchestrated fleets of agents. He points to financial services as an example: 

You might have one agent doing compliance checks, another handling intent detection, and a third pulling records. Agent Bricks lets you manage and monitor those together, not just as silos.

And Shaw reveals that early deployments are already underway:

We’re working with AstraZeneca to build and manage large-scale pipelines using this tool – without needing to write a single line of code.

Managing costs

To that end, Databricks is incorporating cost-performance tuning and FinOps (financial operations) visibility into the platform. Customers can model what an agent deployment will cost – not just one agent, but 100 deployed across 10 departments. Shaw continues:

We support high-performance, low-latency deployments, but we also let customers dial it back if budget is the priority.

What sets Databricks apart in this launch is how closely these features align with its data intelligence vision. Rather than bolt on a chatbot or skin over a model API, Agent Bricks and Lakeflow Designer are firmly tied into the broader Databricks stack – Unity Catalog for governance, MLflow for experimentation, and Delta Lake for reliable data management. As Shaw emphasizes:

This is foundational, not ornamental. It’s a step forward, yes – but it’s also something that builds directly on years of infrastructure we’ve already delivered.

Enterprise buyers are no longer asking whether AI can help. They’re asking how to operationalize it – safely, responsibly, and at scale. The complexity of models, the need for domain-specific training, the demand for trust and auditability – these challenges are not theoretical.

As Shaw puts it: 

We’ve built the foundational pieces. Now we’re automating them. This gives customers everything they need – from model selection to performance scoring – to put agents into production and prove their business value.

This approach also fits into Databricks’ broader strategy of remaining model-agnostic and open. Rather than locking customers into a specific LLM or infrastructure, Agent Bricks supports flexible integrations, including with Anthropic and Meta’s models, and enables agents to be built and deployed inside or outside the Databricks environment.

That flexibility may prove crucial as the AI landscape evolves. As Shaw notes: 

We don’t know where AI will be in two years. But we do know our customers need control, visibility, and trust today.

He concludes: 

This isn’t one agent in a lab — it’s fleets of agents in production, managed with real oversight. Our customers don’t want a demo. They want infrastructure they can bet on.

My take

Ultimately, this isn’t about catching the latest wave of AI hype. Databricks' goal is focused on making good on the promise of intelligent systems that don’t just work in theory but deliver tangible business value – reliably, transparently, and at scale.

This focus on observability and confidence also echoes insights from Dael Williamson, EMEA CTO at Databricks. In a separate interview, Williamson described the real barrier to effective AI as a form of 'corporate archaeology' – where organizations must first excavate, understand, and govern their fragmented data estates before they can deliver on AI's promise. He observed:

Enterprises often don’t even fully know what data they have, where it lives, or how useful it is.

This aligns with Shaw’s emphasis on governance-first design – without clear pipelines, trusted metadata, and observable workflows, AI agents may function technically – but not reliably. Tools like Lakeflow Designer and Agent Bricks aim to close that gap by baking in data quality, lineage, and model scoring from the outset.