Cleared for takeoff - Virgin Atlantic's AI Concierge lands with observability built in

Virgin Atlantic has attempted to put its cabin crew into an Artificial Intelligence (AI) app. Not to replace the crew on its long-haul flights, but to add that same customer experience to the times when a Virgin Atlantic traveller does not have access to cabin crew. The airline was the customer keynote at this year’s Datadog Summit, and we sat down with Mark O’Neill, Senior Manager of AI at Virgin Atlantic, to discuss the role Large Language Model (LLM) Observability had in meeting the business requirements of launching a customer-facing AI product.

Virgin Atlantic currently has a Beta AI chatbot available to travellers, named Virgin Atlantic Concierge. It works across the airline and holiday business lines. Using OpenAI and built in partnership with AI specialist Tomoro, the AI chatbot is part of the airline’s latest app. O’Neill said the airline believes AI will be central to the next evolution of customer experience. That evolution had to be more than just an AI chatbot, he says:

The travel journey can be quite difficult for the customer, and it is not always possible to have a staff member with you, so the vision was how to take the experience of dealing with our people and put it in their pocket.

We know from our customer surveys that people are really happy when they interact with our crew, but there are points in the journey when they cannot, so how do you provide that in those moments? We see AI as an enabler for a better experience.

O’Neill says launching a customer-facing AI product is daunting for brands and shared the two main concerns the airline had. Firstly, it wanted AI to be authentic and described AI tone drift as brand drift. The second was high levels of concern about AI’s ability to hallucinate and give out false information, he says:

We don’t want to be in the press for the wrong reasons.

To manage AI and protect the business, O’Neill says the scope for what Concierge could do was deliberately kept narrow. Reducing the scope, which also reduced the blast radius should the AI make a mistake, he said on the Datadog stage, did not ground the abilities of Concierge. 

Instead, Virgin Atlantic observed its teams and took their behaviour patterns and fed them into the model that Concierge uses. Given the high levels of information we provide to airlines as travellers, Concierge cannot change, or respond to personal identifiable information (PII). O’Neill adds:

We have created explicit system prompts that our brand and marketing team helped with in terms of the right tone and response.

This included understanding the tone and a situational understanding of the traveller. Connection delays, a sudden change of plans, or illness will impact how the traveller communicates with Concierge, just as it would if they were dealing with cabin crew. Virgin Atlantic’s team focused on these situations and has worked on making sure Concierge recognises and adapts its responses to the tone and needs of the traveller.

To prevent Concierge from being used for AI jailbreaking (using prompts to make the AI generate content that is harmful, illegal, or explicit by persuading OpenAI to ignore its safety training), O’Neill’s team built in limits to the length of a user prompt. There is also a rule that Concierge will first check that the query is on topic. He said you can ask it for some football results, but it won’t provide you with an answer. Concierge will also not accept prompts that feature code or JSON queries, or languages other than English. If you ask Concierge a question in French, it will tell you it can only answer in English.

Observability provides safety

These safety features are in part due to the usage of LLM Observability, O’Neill said, hence his role at the Datadog summit.

Observability tells us how Concierge behaves every day.

Datadog LLM Observability was built into the flight plan of Concierge, and O’Neill says it provides end-to-end analysis of every call Concierge makes to OpenAI and its APIs:

On every single interaction between a customer and Concierge, we run a full set of evaluations asynchronously on Datadog. From an operational point of view, it has created a feedback loop for the product teams that is being fed back into the development process.

Additional security has been provided by Pentest Partners.

O’Neill’s story was used at the conference to highlight that as digital leaders release customer-facing AI products, or agents into the enterprise and the operations of the business, the level of complexity increases. As product and development teams become more adept at creating AI outcomes, that complexity could increase dramatically. As a provider of observability platforms, Datadog believes they have an essential component for CIOs who are worried about the customer experience, the resilience of their platforms, data security, data availability, and productivity benefits.

Pejman Tabassomi, Field CTO of Datadog, describes the situation:

Observability helps you understand the model and match that up to performance, to know how many tokens were consumed, the quality and safety of the models, and then to experiment with the models you are using.

Virgin’s next AI destination

Concierge was developed using an enterprise agreement with OpenAI at Virgin Atlantic. The airline has an AI governance board in place, and O’Neill said they are assessing how AI can be used in the airline’s contact centres, both in the UK, in Swansea, Wales, as well as those overseas.

My take

Observability platforms and their providers are well placed to provide some clarity in a fast-paced AI-led world. AI could simplify organisations, but many organizations are struggling with fragmented digital infrastructure and data. Adding, as many must, AI on top of this will increase that fragmentation. As Datadog and Virgin Atlantic portrayed, observability provides organizations with a way to understand this landscape and, from the beginning, understand how AI behaves in the organization and towards your most valuable asset - the customer.