Keeping OnePay Current: How We Built an Agent-Assisted Path to Safer Infrastructure Updates

In The OnePay Platform Story, we shared how a common foundation helps OnePay move quickly while staying reliable and secure. In Inside OnePay's AI Journey, we described AI as a core platform capability, and in Scaling Engineering, we introduced the agents supporting our engineers. This continues that story with Octascribe, the agent that helps us keep the software beneath OnePay current while preserving the controls our customers expect from a financial platform.

The moment a vulnerability goes public, a stopwatch starts, and everyone gets to read the same advisory, attackers included. Patch too slowly and you're sitting on a known weakness. Patch too fast, without understanding what actually changed, and you jeopardize reliability. We need both speed and caution, at the same time, across dozens of Kubernetes clusters spanning test, non-production, production, and isolated PCI environments.

Argo CD continuously reconciles what's declared in Git against what's actually running in each cluster, an operating model commonly called GitOps. Most of what runs on those clusters ships as Helm charts, versioned templates that render into the resources Kubernetes actually applies, and our setup follows a hub-and-spoke model: ApplicationSets pointed at a shared configuration repository, which pins each workload to a chart revision and lets Argo CD reconcile that revision out to the right clusters.

Here's how deceptively small this gets: bumping a chart revision from 0.1.29 to 0.1.30 is often a single line in Git. That one line can also quietly remove a Deployment, tighten or loosen a container's security context, or introduce a Custom Resource Definition that an existing operator has never seen before. The version number is basically a poker face; it gives away nothing. What actually matters is the gap between the manifests Argo CD renders today and the ones it would render the moment that merge lands.

For years, Renovate told us that something new was available, and we still lean on it broadly elsewhere in our stack. It never told us what that update would actually do to us. An engineer still had to read the release notes, translate them against our own configuration, review the rendered fallout, and babysit the change through every environment by hand. Necessary work, sure, but the kind that's always first to get bumped for something louder.

So we built a system with two jobs, not one: move approved versions forward on a schedule, and generate enough evidence, automatically, that a human or an agent can actually see what each move does before it happens.

Promoting Changes in a Controlled Order

Our flow splits two related jobs. First, Kargo, a GitOps promotion engine, spots an eligible upstream version and proposes the matching update to our internal chart or configuration. Once that internal artifact is built and published, Kargo promotes it progressively through the environments where it actually needs to run, writing every step back to Git as a reviewable merge request.

Automatic doesn't mean reckless. Policy-enforced waiting periods make a version earn its way out of non-production before production will touch it, while our monitoring watches for anything that starts behaving oddly. When something urgent turns up, such as a high-priority vulnerability fix, authorized engineers can put a foot on the accelerator without ever skipping review.

That's a controlled path for delivery, but a merge request that just says the chart revision moved still doesn't explain whether the change is safe to promote. For that, we need to see what Kubernetes will actually receive.

Turning Rendered Manifests Into Evidence

Octadiff is a service we built to generate diffs using the same rendering infrastructure that drives our clusters. When a merge request opens or updates against the configuration repository, each affected hub's Octadiff instance renders the complete manifests for the current and proposed revisions, compares them, and posts the results back to the merge request: a per-hub view of every affected resource, down to the ConfigMap tweak, the RBAC change, the CRD schema shift.

That's what a reviewer actually needs to make a call, but it can run long. A large upgrade can produce thousands of lines of manifest diff alongside release notes and compatibility requirements. Generating the evidence solved one problem. Reading all of it, consistently, every time, was the next one.

How Octascribe Analyzes Risk

Octascribe is a custom orchestrator agent, built on the Kagent framework, that splits manifest, release, compatibility, and policy analysis across focused specialist agents instead of dumping everything into one model's context. Depending on the change, those specialists might inspect Octadiff's results, reason through Kubernetes manifests, read Helm metadata, check Kubernetes API and version compatibility, pull upstream release information, or validate Argo CD project configuration, then hand back structured findings rather than raw output. The analysis runs in stages, so later checks can build on what earlier ones already found.

Octascribe then posts a consolidated report to the Kargo merge request: what changed, what looks risky, which checks completed, and where a human still needs to look. For eligible low-risk changes, it can request auto-merge, though GitLab's own protected-branch, approval, and CODEOWNERS rules still get the final word. Octascribe never merges code or touches a running cluster directly, and any promotion targeting production still needs an engineer's sign-off.

Models sometimes go quiet. Networks hiccup, external services time out, an analysis finishes without all the evidence it needed. We treat that as a blocked analysis, never as an implicit approval. The promotion stays stopped until the team investigates or finishes the review by hand.

Designing for Untrusted Inputs

Running this analysis autonomously means treating everything Octascribe reads, release notes, merge request content, rendered configuration, as potentially untrusted data rather than instructions. VCS webhook traffic is restricted at the network edge, and an intermediary builds the agent invocation from a controlled template instead of forwarding raw webhook text as if it were a command. Octascribe's analysis tools are deliberately non-mutating: they can inspect source material, rendered manifests, and platform state, but they're never handed tools that merge code or change running clusters. The service keeps only narrowly scoped write access for its own execution state, nothing more.

None of that makes prompt injection or a bad model call impossible. It narrows the paths through which untrusted content could influence the system and limits the damage if an analysis behaves unexpectedly, with human approval for production standing as an independent backstop. It's the same principle behind the agent platform we described in Scaling Engineering: agents inherit narrowly scoped access, reusable controls, and observable tool boundaries, rather than getting broad authority just because a workflow happens to be automated.

Analysis Before Promotion

Octascribe's job starts before an artifact ever enters the environment progression. The moment Kargo spots an eligible upstream release, Octascribe can size up that release and its likely infrastructure impact before the version gets promoted through our internal system, which helps separate routine changes from the ones that actually need migration steps or closer engineering attention. Paired with the rendered-manifest analysis that happens later, it gives us evidence at both ends: what the upstream project says it changed, and what our own environment would actually run.

What We Learned

Control model context. Large merge requests can quickly fill a model's context with generated manifests and release information. As context grows, analysis can become less reliable or reach hard limits. We isolate data-heavy work in focused specialists and return only relevant, structured findings to the main analysis. This improves both efficiency and consistency.

Make handoffs structured, but not rigid. At one extreme, agents can be left to decide what matters with little guidance. At the other, every handoff can become so constrained that the system cannot adapt to an unusual change. We use skills and defined output formats to make specialist results predictable while preserving room for the agent to follow the evidence.

Use only the pathways a change needs. An internal chart promotion and a new upstream release present different questions. Kagent's skill model lets Octascribe select the relevant analysis path instead of running every specialist for every change.

Build feedback loops around the agent. As we found while building Tokki, the quality of an agent depends heavily on the environment, tools, and feedback around it. Octascribe works because it receives concrete rendered evidence, operates through purpose-built analysis tools, and stops when required checks do not complete.

Design for untrusted data and partial failure. Security controls and failure handling are part of the product, not safeguards added after the automation works. Restricted ingress, controlled invocations, non-mutating analysis tools, staged promotions, monitoring, and human production approval work together as independent layers.

Keeping the Platform Current

Octascribe cuts down the recurring manual work of reading release notes, interpreting Helm diffs, and evaluating compatibility across environments. Engineers get to spend their attention on exceptions, real risks, and the one decision that still has to be human: the production call, instead of reassembling the same evidence by hand every time.

This isn't a system we finished and walked away from. The infrastructure we run, the software we depend on, and the models available to us keep changing, so we keep refining the agents, skills, controls, and evaluation criteria as the platform evolves. Keeping our platform current is quiet, unglamorous work, but customers feel the result directly: a service built to stay secure, reliable, and ready as OnePay grows.

If you think in GitOps, agent architectures, security, or the operational discipline that makes all three trustworthy, we'd love to hear from you. Check out our open roles at onepay.com/careers.