Share
In this Article
Share
At OnePay, our vision is to create the one place to money. To truly serve millions of customers sustainably, we have to solve a classic operational paradox: as the business grows, we must be able to handle exponentially growing customer contacts and achieve maximum efficiency, without compromising on quality of service.
In the first post of this series, Inside OnePay’s AI Journey: From Infrastructure to Experience, we outlined three areas of focus: Operations, Productivity, and Product. In this post, we delve into the first pillar: AI in Operations.
AI Beyond Chat Bots
When AI in Customer Support comes up in conversation, our minds immediately go to Chat Bots. We’ve all experienced them in one place or another, and have had varying degrees of satisfaction interacting with them. Generative AI offers a unique opportunity to create natural conversational experiences, and we fully take advantage of it, but our strategy goes much deeper than the bot.
We took a step back and looked at the entire lifecycle of a customer contact, from the time they initiate a contact, to all the post contact steps required. We mapped where AI can help make each leg of the lifecycle more efficient, ensuring the appropriate feedback loop for product enhancement and quality controls are implemented.
The Lifecycle of a Contact
We break down the lifecycle of a contact into three interconnected stages. The first, the pre-contact stage, is where a customer initiates a contact, whether by calling or chatting with us. The second is the during-contact stage, where a customer is chatting live with a customer support representative. The third and final stage is post-contact. This is the stage where product analytics, quality assurance and other steps are performed.
We have an opportunity to improve our speed of execution in each of these stages. Before and during contact, the objective is to resolve customer issues and get them back to their daily lives as quickly as possible. Post contact, the objective is to provide product feedback and discover quality gaps and resolve them promptly. The remaining question is, can Generative AI help achieve maximum speed and efficiency here without compromising quality of service?
Meet the OnePay Operations AI Team
We have developed and deployed five specialized AI Agents across the three phases of the customer support lifecycle. Meet our Operations AI team.
Chat Agent
Previously, we deployed a classic, rule driven chat bot - a bot that understands natural language to some degree, tries to distill the “intent” of the customer from each turn of the conversation, and maps it to a pre-defined set of “supported intents” the bot recognizes. For each supported intent, the bot is configured with a deterministic workflow, parts of which might require more inputs from the user. This results in a maze of complex processes, and a fragile bot with basic natural language capabilities trying to determine intent based on minimal context.
Enter Chat Agent, the first of our AI agents that operates in the pre-contact stage. The key advantage we sought is leveraging generative AI to have a superior natural conversation with a customer, a more accurate way to derive intent based on a full conversation context, and a more manageable way to support the ever-increasing support procedures. But before we get there, there were a few fundamentals that had to exist in order for this agent to thrive.
Knowledge Base: when a new representative joins the team, they require detailed training and a reference guide they can refer to when on-the-job that contains the various support procedures. Similarly, AI needs a knowledge base to gain the same context. This technique, Retrieval-Augmented Generation (RAG), is commonly used to help ground models in the context of a use case. Given the velocity of our product development, it was paramount to create an automated, self-refreshing knowledgebase that is constantly up to date based on the latest features shipped.
Tools: in order for the AI to execute on the procedures it’s trained on, it needs to be able to access certain parts of the system, based on the procedure it’s executing. For example, if you ask it what your balance is, it needs to be able to use a “tool” that can fetch you users balance.
Policies: we realized that "prompt engineering" wasn't scalable enough. We needed to allow product and operations staff to manage the AI agent. We built a framework for Policies, a configuration layer that allows product managers to define tone, format, and guardrails without touching code. This decoupled logic from language, allowing for rapid iteration on the agent’s personality.
Evaluations: Given the nature of generative AI, and the pace at which model development is moving, we knew we needed to tune prompts, policies, models and providers on a regular basis. We needed a key capability - a fast, automated way to re-evaluate the behavior of the agent when changing any of these variables. We developed an evaluation framework accessible beyond just engineers, and we use it across all our AI agents. We leverage a combination of deterministic criteria and the LLM-as-a-Judge model for these evals. By making these self-serve for product teams, we’ve unlocked rapid change and testing loops where our Product Team can modify prompts and re-execute evaluations, without requiring engineers' involvement.
Controlled Rollout: We released the generative responses in shadow mode, allowing us to see what the Chat Agent would’ve said in each situation if it was live. Once we tuned and built confidence, we released it to a small percentage of contacts and scaled gradually, all while improving our agent’s outputs. We continue to use these controlled rollouts when we make significant changes to the Chat Agent.
Phone Agent
Given the overwhelming impact we saw in Chat Agent, and with the advancement in Voice AI in the last 12 months, we saw an opportunity in bringing the foundation we built for Chat Agent to the phone line. We introduced Phone Agent, the second of the pre-contact agents. It responds to customers who call us, and tries to help them in a similar way to Chat Agent in the chat channel.
It was remarkable how quickly we were able to bootstrap Phone Agent based on the existing AI foundation we built. However, there are unique challenges specific to voice agents. For example, in a chat conversation, it is common for Chat Agent to include a link in the response that leads the customer directly to a specific part of the app. On the phone, that had to translate to step-by-step instructions that our Phone Agent narrates back to the user.
Co-Pilot Agent
There will always be a percentage of contacts that require support from a customer support representative. When a representative connects with a customer, they need to very quickly gather context on the customer account, the current issue they are facing, in an environment where the only constant is rapid product velocity. Co-Pilot Agent operates alongside each representative in every contact. It quickly summarizes the conversation that already occurred between the customer and Chat or Phone agent, gathers key information about the customer’s account, contact history and other key contextual attributes about the user. It follows the conversation between the customer and representative, and suggests possible solutions or resources to the representative. The objective is to resolve the customer’s issue as quickly as possible, not just provide quick search results. We’re actively working on rolling this out across our Operations teams, and building towards making this useful for back-office operations.
Quality Assurance (QA) Agent
Meet the first of our post-contact agents. Traditionally, a manual QA program can only afford to review a small number of contacts using random sampling or other metadata. At high scale, the sample rate gets smaller and often becomes less statistically representative of quality themes. We developed the QA Agent to be able to review and grade every single contact after it ends. QA Agent looks for key quality signals across thousands of contacts every day. As it turns out, this agent became a great coach. It generates individual coaching summaries for representatives, highlighting areas they do well and areas of improvement. Providing timely feedback to our team allows us to continuously improve the customer experience and satisfaction.
Analytical Agent
As the name suggests, the Analytical Agent summarizes contacts, extracts invaluable information from unstructured data, looks for patterns and aids in product analytics and adhoc troubleshooting. It closes the loop by informing the product roadmap of critical patterns in support cases. The objective is fast insights and a tight loop to continuous improvement.
We analyze conversation transcripts via a series of model calls to create a narrative: user intent, agent actions, call conclusion, and resolution status. We then correlate this structured data with agent actions and user account status. These outcomes serve as signals to direct deeper QA agent investigation or to prompt LLMs to identify themes across many contacts.
Conclusion
As mentioned in the Platform Story, we believe in building platform infrastructure that encourages reuse, accelerates progress and improves consistency. The key to unlocking the launch of these agents with speed and confidence was a result of investing in the underlying platform. A self-refreshing knowledge base, a policy management system, an automated evaluations framework, a pluggable backend into different LLM providers and model versions.
The results are clear — in 2025, our Phone Agent increased automated resolution of customer issues by 60%, and Chat Agent by 20%. And we’re only getting started.