Design for testing Part 2: ​​Business vs. application logic

Capital One is on a journey to evolve our testing maturity by automating all testing and moving toward a continuous deployment model. Along the way, we’ve discovered that test maturity can be developed by separating concerns in some key ways. This is part 2 of a two-part article exploring these principles. 

Part 1 introduced design for testing: Apply layers and dependency inversion to achieve high cohesion and loose coupling. This approach isolates business logic and workflows, enabling fast, local unit testing on every developer’s machine.

In Part 2, we differentiate between two types of functional requirements: business logic and application logic. Business logic is domain-specific and technology-independent, while application logic arises from technology choices. We will explore how tests can be categorized under business or application logic and which testing technology integration is best suited for each development phase.

The test automation pyramid, introduced by Mike Cohn back in 2009, illustrates the correct proportion of unit tests to higher-level tests in the shape of a pyramid.

The wider base of the pyramid represents a larger number of unit tests than the narrower top, representing higher-level end-to-end tests or “UI tests.” The reason Mike Cohn posited this test distribution is that higher-level tests are “brittle, expensive to write and time-consuming”; therefore, we should limit how many of those tests we have. The balance between these tests provides us with pragmatic guidelines so that the barrier to entry for test automation remains low. In Mike Cohn’s words, “Automated tests provide cheap insurance.”

The two main points distilled in Martin Fowler’s blog “​​Practical Test Pyramid” (by Ham Vocke) on the test pyramid help to sum this up:

1. We should have different test sizes, where size is the number of system artifacts being exercised.

2. To maximize the “return on investment” of automated tests, we should maintain a larger number of small tests and limit larger ones.

Another way to classify tests is by the type of requirement they test: functional or nonfunctional. Within functional requirements, we have two subgroups: business logic and application logic. Business logic (or domain logic) requirements are those that are inherent in the business itself. Application logic requirements are those that we accumulate as we make software design choices. We impose these requirements on the system with our choices on tech stack, ways of integration, data structures and so forth.

With tests broken down by requirement types, we can organize our tests by considering the following three dimensions:

1. When: Which stage of the Software Development Life Cycle (SDLC) you’re currently in

2. What: Which of the three requirement types you should focus on

3. How: Which test method to use

The test cube tells you which test method to use, given the SDLC stage you’re in and the type of testing you need to do. For instance, business logic requirements can be tested in their entirety with unit tests at development time on the engineer’s local machine. These tests are executed again at a pull request (PR) submission just before code merge is performed. The ability to test all of our business requirements early in the software life cycle is a tremendous advantage to have. The successful execution of all business logic tests provides the confidence needed to move forward in the development cycle. Additionally, engineers can start the first phase of testing application logic with contract testing and some component testing with mocks on their local machines.

At Capital One, our engineers and product managers (PMs) work together to form the business requirements around software needs and to codify the requirements as executable tests. Using this methodology helps ensure that we have “behavioral” coverage in our tests by removing the practice of simply handing off requirements from a PM to the tech team. Our SDLC has recently formalized this collaboration through the use of Jira Xray, a shared “test management application” that allows both engineers and PMs to capture and track requirements and tests together.

Once you begin deployments into remote environments, you also increase the breadth of your application logic tests. The first of the remote environments is the integration environment. Since other systems will also deploy into this shared environment, you’ll be able to run live integration tests with those systems. However, this environment is expected to be unstable, as those systems might be down due to more frequent deployments. If the instability of the environment becomes a hindrance, there is the option to use virtualized versions of those service APIs instead.

Much of the instability should be removed when we enter the quality assurance (QA) environment. The QA environment should mirror the production environment as much as possible. This is where we run our final end-to-end tests to prove out the system as a whole. We can also run nonfunctional tests here as long as data and processing loads mimic what is experienced in production.

Testing occurs in production more often than most would expect. First, any form of blue-green or canary deployment is a type of test we run in production. Nonfunctional tests, such as chaos testing, are executed in production at both scheduled intervals and random times.

A quick word on development and the testing we do specifically for development, which I will refer to as “development testing.” Development testing is a set of activities a software engineer does while coding. These tests do not need to be part of a shared unit test suite. Suppose we are using Spring JPA and we have a find-by method with a customized @Query annotation:

Making sure a database query returns a valid result is a development task and does not need to be part of a unit test suite that runs repeatedly. If we have separated our business logic from application logic, then having a unit test for such things as data access to a database is unnecessary. Moreover, open source libraries have done away with common boilerplate code by reducing the code necessary to implement it, as shown above. In these cases, we do not want to conflate testing our application with testing these libraries.

To bring this home, I will draw an analogy between software development and car repair. When putting an engine together, mechanics use torque wrenches to help them precisely apply a set amount of torque to fasteners. There are specific foot-pound torque values and a prescribed order in which bolts are tightened. The torque wrench is a “testing” tool, and it lets the mechanic know if they have installed the part to specifications. When we use a JPA annotation or the AWS SDK to interact with external parts of our system, we also need tools to help us test if we have installed those parts of our tech stack correctly. But, instead of torque wrenches, our toolboxes are filled with Testcontainers and AWS SAM Local instances.

Flipping back to the car analogy, imagine taking your car into the shop for maintenance. Your mechanic might run high-level diagnostic checks on the car. These are high-level tests that help the mechanic quickly evaluate the condition of the car and whether it is fit for the road. These tests can point to underlying issues.

Running diagnostics is a practice analogous to the tests we run in our pipeline. We run high-level tests with live dependencies to confirm the production-ready condition of our system. And just as we wouldn’t expect a mechanic to test every bolt in a car with a torque wrench during these inspections, we should not expect our pipeline tests to spin up Docker containers, in-memory databases and the like so that we test every “bolt” in our system. The cost of automating such tests and keeping them up to date will outweigh the benefits they provide.

Finally, last but not least are any exploratory tests we conduct with human testers using the system. Exploratory tests fill in gaps we have in our automated testing suites. There may exist test scenarios where automation is imperfect or overly costly. In those handful of situations, conducting a manual test is more practical.

We should prioritize testing application logic with live dependency tests rather than unit tests. Attempting to cover application logic requirements with unit tests to any great extent with mocks or virtualized dependencies will result in a low return on investment. The reasons for this can be boiled down to the following:

1. To test the application logic, our tests will require many mocks. Setting up these mocks can quickly turn into an expensive endeavor. Mocks usually need to change whenever code changes, which makes for a brittle set of tests.

2. Test results will be based on mocks, which are less accurate representations of actual systems.

As we progress in our testing maturity, we quickly find that “testing” is a forcing function to design software with “separation of concerns” in mind. Testing maturity is equal parts having advanced tooling and adopting strong design principles. In Part 1, we touched on designing for high cohesion and loose coupling with the use of layers and dependency inversion across these layers. This gives us the ability to isolate business logic and business workflows so that they might be tested entirely with unit tests that are fast to run and can execute on every developer’s local machine.

In this second part, we recognize that functional requirements come in two flavors: business logic and application logic. The business logic is inherent to the domain space and is independent of the tech stack used. In contrast, application logic stems from our technology choices. Testing how well our technologies are integrated should be done in the integration or QA environments. Extensive testing of application logic outside of such environments requires a massive amount of mocking acrobatics that result in large development costs. Mocking also sacrifices encapsulation, a key design quality, since test mocks need to know the internals of what it is mocking. Such overuse of mocks binds the test apparatus to the internals of the application, making the test suite brittle.