Scaling enterprise testing on Kubernetes with Argo

Every engineer knows the quiet anxiety of watching a CI/CD pipeline run. You commit, you wait for the build and then you wait some more. Now imagine that at enterprise scale: thousands of test agents running at once, across dozens of teams, all validating business-critical services on the same shared infrastructure. That’s the problem our platform exists to solve, and for years, the way we solved it was quietly holding us back.

At Capital One, we’re a technology company that happens to operate in one of the most tightly regulated industries there is. That combination raises the bar: our internal platforms strive to be fast and safe, to scale for many teams at once and to prevent any single workload from destabilizing the rest. So when our first-generation testing pipeline started hitting hard limits, we didn’t just move it; we rebuilt it on Kubernetes using open-source, Cloud Native Computing Foundation (CNCF) projects from the Argo ecosystem.

In this post, we’ll walk through three things:

• Why we moved off our first-generation pipeline
• How Argo orchestrates a single test run, end to end
• The guardrails that keep the platform stable when thousands of tests arrive at once

The headline isn’t just “we moved to Kubernetes.” It’s that we learned to run massively parallel test workloads without letting any single burst threaten a shared, multi-tenant cluster.

Our original platform got the job done, but it was built from a highly complex set of managed services linked together by a proprietary orchestrator. Execution logic lived inside cloud-specific state-machine definitions, far away from the containers actually running the tests. Three pain points stood out:

• State-machine sprawl. Changing execution logic meant editing large, cloud-specific workflow definitions. The orchestration lived far away from the code it was orchestrating.
• Cold-start latency. Every test task provisioned fresh capacity and pulled container images cold before doing any useful work. That overhead was charged to every run—even the smallest smoke test.
• Redundant resource consumption. Frequent, repetitive downloads of container images and code repositories across parallel tests created unnecessary strain on shared infrastructure.
• Debugging with limited observability. When tests ran in execution contexts with limited observability, troubleshooting a stuck run meant coordinating live debugging sessions and bolting on extra logging just to see what was happening.

We wanted something declarative, Kubernetes-native and self-cleaning—and we wanted to build it on open source so our patterns could scale across the whole company rather than being locked to one team’s bespoke tooling.

Instead of extending our proprietary orchestrator, we moved our execution layer onto a dedicated Amazon Elastic Kubernetes Service cluster and adopted three battle-tested projects from the Argo ecosystem. Each owns a clear slice of the problem:

• Argo CD manages the platform itself. Using the GitOps “App of Apps” pattern, every component, including event sources, workflow templates, access rules and node configuration, is declared in Git and continuously reconciled onto the cluster. The platform’s desired state is version-controlled, reviewable and reproducible.
• Argo Events ingests work. It runs as a native, event-driven framework inside the cluster and long-polls a message queue for incoming test requests, using workload identity, so there are no static cloud credentials stored in the cluster.
• Argo Workflows is our Kubernetes-native state machine. Execution pipelines are declared as YAML templates instead of cloud-specific scripts, and they run as first-class Kubernetes objects we can watch, log into and debug like anything else in the cluster.

Adopting these projects allowed us to remove an entire category of maintenance and gave us a foundation the rest of the organization already understands.

So what actually happens when you kick off a test? Here’s how a single request flows through the platform.

1. Ingestion. A lightweight front-end API validates the request and drops a small payload onto a message queue. Inside the cluster, Argo Events is already watching that queue. When a message arrives, an event sensor fires and instantiates a workflow. Because the poller authenticates with workload identity, which eliminates the need to store static credentials, this significantly reduces the risk of credential exposure.
2. Provisioning an isolated environment. The workflow’s first job is to create a dedicated namespace for the run, complete with its own service account, minimum-privilege role-based access control and dynamic configuration objects carrying the test parameters. Every run executes within an isolated environment to limit the potential blast radius.
3. Parallel execution. The workflow fans out and launches the test suite as Kubernetes jobs run in parallel. The test agents are runtime-agnostic, driven by the request payload rather than hardwired into the platform, so the same machinery runs many different language ecosystems.
4. Self-cleaning life cycle. When the run finishes, Argo Workflows exit hooks fire automatically. Results are reported back to our control-plane API, container logs are archived to object storage and then the entire temporary namespace is deleted. Because Kubernetes reclaims resources natively at the namespace level, we have a low risk of orphaned resources, dangling configuration or lingering test state.

Getting tests to run on Kubernetes was the easy half. The interesting engineering that we’re proudest of was making the platform stable when thousands of test agents show up at once on infrastructure shared by many teams.

Raw parallelism is a double-edged sword. The same fan-out that makes runs fast can also stampede a cluster: a large burst of requests can flood the control plane, saturate scheduling and degrade the experience for every tenant — not just the one that triggered the surge. That’s why we performance-tested the platform to understand its limits and where it strained. We then used those findings to drive a set of deliberate guardrails.

Rate-limiting at the front door. The most impactful change came at the event-source layer. Instead of letting Argo Events pull work as fast as the queue could deliver it, we bounded consumption at the sensor, capping how many messages are pulled and processed concurrently, and using long-poll batching to smooth spikes. This turns a chaotic flood into a steady, predictable stream the cluster can actually absorb. 

We found that for enterprise scale, parallelism is easy, but predictability is hard. To build a system that survives thousands of concurrent tests, we designed for three stability principles:

1. Ingestion smoothing: Never pull work faster than the system can process it (rate-limiting).
2. Tenant fairness: Isolate high-resource workloads to prevent noisy neighbors (node pool isolation).
3. Queue-as-pressure-valve: Let requests wait as pending objects rather than dropping them or crashing the scheduler.

A tuned control plane. We bounded the workflow controller’s concurrency and API throughput, and set aggressive time-to-live cleanup so completed workflows won’t accumulate. This helps the controller stay responsive under sustained load.

A global cap on concurrent workflows. Argo Workflows lets us set a hard ceiling on how many workflows execute at any one time, and this turned out to be one of our most valuable safety valves. When more requests arrive than that ceiling allows — say, a batch of long-running suites is holding slots during peak hours — the extra workflows aren’t force-scheduled onto an already-busy cluster. They simply wait: Argo keeps them queued as pending Kubernetes objects (their state persisted as native objects in the Kubernetes cluster storage) and admits them automatically as running workflows finish and slots free up. That one setting protects the cluster’s shared dependencies (databases, downstream services and the control plane itself) from being overwhelmed when everything lands at once. Nothing is dropped; work is just paced to what the cluster can safely handle.

Isolating heavy runs by node pool. Not all runs are equal. A small suite and a massively parallel one have very different footprints, so we route them differently. Using Kubernetes RuntimeClass (a way to select the runtime and placement for a pod) together with node-pool selection, large parallel runs are steered onto dedicated capacity, keeping a big run from crowding out everyone else on shared nodes. Where to draw that line wasn’t a guess: we set the threshold from real distribution data, tuning it to the point where run sizes naturally cliff.

Pre-warmed capacity. To attack cold-start latency, we keep a pool of nodes warm and ready ahead of demand, and lean on node-local image reuse so agents don’t re-pull images that are already present. Scheduling feels close to instantaneous compared to provisioning capacity on the critical path of every run.

Taken together, these guardrails are the difference between “it works in a demo” and “it’s a stable, multi-tenant platform.” Parallelism gives us speed; rate limits and isolation give us predictability, and predictability is what a shared enterprise platform actually needs.

One recurring challenge in enterprise testing is identity: a test often needs to act as a specific role to reach the resources it’s validating. We leverage native Kubernetes-to-cloud workload identity, where a pod’s service account maps to a cloud role and temporary credentials are injected transparently as the pod starts. To the test container, it simply has the right identity, so developers write standard cloud SDK calls with zero assume-role plumbing.

This gets more powerful for cross-account testing, where a run executes in our central cluster but needs to reach resources in a separate target account. Instead of writing custom role-assumption code into our agents, we leverage a cross-account pod identity association for a fully native implementation. A single association links the pod’s service account to both the platform’s source role and the target account’s narrowly scoped role. The platform’s identity agent then injects temporary credentials for that target role directly into the pod. The customer’s side of the trust is deliberately simple; provisioning one tightly scoped role that trusts our platform is all it takes. The result is the same everywhere: standard identity, no custom credential code and no need for long-lived secrets to cross account boundaries.

Trading a complex, proprietary pipeline for an open-source, Kubernetes-native design paid off on several fronts:

• Lower scheduling latency. Pre-warmed nodes and node-local image reuse make execution feel near-instantaneous compared to provisioning capacity per run.
• No more debugging with limited observability. Because runs happen in a cluster we operate, engineers have real-time access to pod logs, life cycle events and workflow state. Troubleshooting a stuck test takes seconds, not a scheduled call.
• Simpler, declarative pipelines. Execution logic lives in reviewable YAML templates in Git and reconciled by Argo CD, not in bespoke, cloud-specific state machines.
• Self-cleaning by design. Namespace-scoped life cycles mean resources clean themselves up, with little risk of orphaned states.
• Stable under load. Sensor-level rate limits, a tuned control plane and node-pool isolation let us run at enterprise scale without one team’s burst degrading everyone else’s experience.

Open source and Kubernetes-native design gave us a platform that’s simpler to reason about, easier to operate and resilient under real enterprise load. The Argo ecosystem lets us stand on the shoulders of the broader CNCF community instead of maintaining bespoke orchestration, and the patterns we’ve built are portable across teams rather than locked to ours.

By treating our testing platform not as a bespoke service, but as a standard collection of Kubernetes-native components (Argo CD, Events, Workflows), we’ve moved away from maintaining a complex, proprietary stack. For our engineering organization, this architecture serves as a blueprint that helps us balance security, stability, and developer agility. 

If you’re building on Kubernetes, the takeaway we’d pass along is this: parallelism is easy to celebrate and easy to under-engineer. Design for the shared cluster from day one, and never let speed come at the cost of stability. To see more of how we build with and contribute to open source, explore Capital One's open-source work.