Best practices and top trends for serverless at scale

Capital One’s journey to serverless architecture taught our teams a lot. The following are what we consider to be serverless best practices so others can leverage the potential of serverless computing. This blog will also discuss lessons learned, top serverless computing trends and predictions for serverless adoption.

Capital One operates within AWS Lambda at a massive scale. We run thousands of accounts, and within those are tens of thousands of Lambda functions. Also, Capital One is a bank and is highly regulated, so we have to create compliance controls around how we operate all these accounts. What do you do when you need to manage thousands of functions? You create standards. 

Setting development and compliance standards is imperative. That’s why we launched our Serverless Center of Excellence (COE) to support our cloud adoption and modernization and to drive our serverless strategy, enablement and support. We believe our COE defines the best practices and standards for leveraging serverless architecture at scale.

At Capital One, we have 10,000 engineers across multiple lines of business. While I work in the Retail Bank line of business, there are other engineers in the Credit Card line of business. There are still additional engineers in Cyber. If I don’t engage the other lines of business, I risk making siloed decisions that could impact the whole enterprise. Decisions that impact the entire enterprise should be collaborative. Our Serverless COE team includes representatives from each line of business who collaborate to establish best practices for serverless at scale. 

We not only get rid of siloed decisions, but our COE also reduces tech debt. Our team meets regularly to discuss and create guidance on all kinds of enterprise issues, such as Lambda default settings, observability, the runtime deprecation process and training. This ensures we can scale serverless responsibly across the entire organization.

Our adoption of serverless across the enterprise has given us hands-on experience and invaluable lessons. Here are the key takeaways.

Leverage industry-standard tools, particularly those designed for serverless, and empower developers to be productive and iterate quickly. Tools like AWS Serverless Application Model (SAM) help streamline development processes. AWS SAM has a command-line interface that allows you to build and iterate locally without needing to deploy to the cloud.

Shift the unit of scale away from requests per second (RPS) to concurrency and make sure your developers understand this new paradigm. This might be a new concept to your team, so you’ll want to make sure that your developers understand how to calculate concurrency. Concurrency equals the average requests per second multiplied by the average request duration in seconds. Everything is measured in seconds.

Let’s look at an example. Say you have a Lambda-backed API with 100 RPS. Let’s assume it runs your function for an average of 500 milliseconds—half a second. 100 times 0.5 equals 50. It’s going to consume 50 units of concurrency. How about one more example? RPS starts at 100, but your durations are now higher—two seconds. That’s 100 times 2, so now you need 200 units of concurrency to service this use case. Always keep in mind that if you’re working in Lambda, the goal is to drive the invocation times down and minimize the amount of concurrency consumed. Keep in mind there’s a limit to the amount of concurrency available per account per region.

Embrace a multi-account strategy where each team, or each team and environment, has its own AWS account. Opt for many small accounts over a few large ones in order to limit the blast radius and allow teams to scale usage with fewer constraints. Why is having multiple smaller accounts better than having a few large accounts? Because this strategy allows you to limit the blast radius and provide each team with its own accounts. AWS Organizations makes governance and management of these small accounts a lot easier.

Set development standards. Establish clear standards, like the use of aliases and versions, to ensure consistency across functions—and be sure to enforce these standards. For example, we require a (LIVE_TRAFFIC) alias to be the target for all Lambda functions. This ensures that we understand how all Lambda functions are executed. We also require all functions to be deployed using versions to ensure consistency and prevent accidental changes.

Through our COE, we’ve identified a list of recommended serverless best practices that come from many trials and much testing. Here are the top four.

• Optimize memory. Memory is the one setting that has a big effect on function performance and cost, so getting it right is important. Utilize tools like Lambda Power Tuner to optimize memory allocation.

• Optimize for Lambda’s cold/warm life cycle. This includes not only code but also packaging and configuration optimizations.

• Understand observability signals. The need to observe your Lambda applications doesn’t change, but the tools and techniques might. Keep an eye on key metrics to ensure optimal performance and resource utilization.

• Optimize for cost and performance. Use metrics to understand how to allocate resources. This understanding will result in lower cloud costs and more performant serverless applications.

Here are the top trends I see in serverless technology. 

• Multi-cloud approach. There is a growing interest in using a multi-cloud approach in serverless computing. The benefits of multi-cloud deployment for serverless applications are that there is no vendor lock-in and it increases availability. 

• Use of AI. AI models can be deployed via serverless architecture. This can prove efficient for scaling machine learning tasks.

• Securing the software supply chain. Third-party dependencies form an attack vector for serverless applications. Securing the software supply chain is even more significant. 

• Durable execution. Serverless developers are looking for ways to ensure code runs to completion. This becomes vital as businesses move critical business processes to serverless environments.

I predict that the interest in serverless architectures will continue to grow, and large enterprises will continue to realize the benefits of lower total cost of ownership (TCO), allowing developers to focus on delivering value and delighting customers. 

If you are interested in more of our learnings, check out my team’s serverless blogs. Feel free to borrow from these best practices if you and your team are also seeking to leverage the potential of serverless computing within a massive-scale environment.

New to tech at Capital One? We’re all in on the cloud, serverless and open source.

• See how we’re building and running serverless applications at a massive scale.

• Explore open tech jobs and join our world-class team in changing banking for good.