Enterprise testing: Challenges, benefits and best practices

The legendary UCLA basketball coach, John Wooden, is famous for winning 10 championships in 12 years. However, his approach to practice is one aspect of his career that is often overlooked. During the first day of practice, you are not taught how to dribble or shoot the ball, you are taught how to wear your socks and tie your shoes. His reasoning was simple—-no matter how talented a player is, a badly worn sock or shoe can adversely affect their jump shot or lead to a serious injury. I found this analogy to be applicable in software engineering. No matter how talented your algorithms are, if its "socks" are not properly worn, bugs might get in and crash it. By using "WH questions", this article will attempt to decipher the challenges facing testing as a feature in most enterprises today.

What is the value of software testing? Analyzing costs and preventing bugs

The de facto answer to this question is “how to find bugs before our customers do?” Another way to derive an answer to this question is to explore the cost of finding a bug in the software development cycle. 

Using the context that time and money are directly proportional, the graph above shows that the earlier a bug is found in the development life cycle, the cheaper it is to fix the bug. For example, when a bug is found during integration testing in QA, the fix requires having to iterate and spend time in the previous steps. However, this is way cheaper than fixing a bug that made it to production and adversely affects customers. An answer to the question: can you test to reduce the total cost of ownership, TCO, for the application we are building?

Market competition, application stability, and time constraints with testing

Given the narrative from the previous section that testing helps to reduce the total cost of ownership and cost reduction is one of the biggest drivers of tech transformation...this begs the question why is testing not considered as a feature for every project instead of as an afterthought?  A widespread sentiment about testing is that it is a good practice, but it will take time to implement. Time is a scarce but highly valuable commodity in any enterprise, let alone IT. Writing unit tests takes extra time as opposed to writing only executable code, (stay with me....I am not advocating this practice...please read on). Setting up acceptance tests, functional tests, performance tests and integration tests all takes extra time. Another reason some teams don't embrace testing, is the culture in some organizations that rewards first to market and penalizes teams that spend that extra time producing a more stable application. 

An accepted answer to the question of "why don't companies test?" is that it will take extra time and prevent delivery of a product ahead of schedule or being first in market.

Determining what to test

In the previous two sections, I have detailed the two main schools of thought when it comes to testing. No matter which camp you belong to, there is some sort of consensus that some level of testing is required for any non-trivial software application. What do we test? Everything? Something? Nothing? Like most things in life, the answer depends and it is somewhere in the middle. As stated in the first section, the earlier you find a bug the cheaper it is to fix. It does not get any cheaper than finding bugs in the requirement gathering stage. Yes, the software application requirement document is the first thing we should test. Getting well defined requirements and preventing scope creep goes a long way in weeding out potential bugs. With respect to cost and time here are aspects of a software application that should be tested at a bare minimum:

• Software Application Requirement Document
• Non-boiler plate executable code
• Integration Testing
• User Testing

The testing pyramid: The correlation between testing approach and scope

Testing is not a monolithic concept that blankets all software application components. It should be viewed and applied with a contextual lens based on the component in question. This means bringing it to the forefront at the beginning of each phase. A common refrain is, "We will add testing AFTER we go live." Although well intentioned, other priorities end up taking precedence over testing and eventually it becomes forgotten. Besides, it has been well-documented that it is much more difficult and much more time-consuming to include testing after an application is in full production mode. How we test is directly correlated to what we test, as depicted in the table below.

The table above is a subset of the testing pyramid most organizations use to determine what to test. As you go up the pyramid, the amount of time spent in testing increases. The amount of time to set up these environments also increases. These time investments dissuade some organizations from implementing a testing framework for their applications. Large enterprises have teams that build automated tools that do all the undifferentiated heavy lifting for application teams, and this helps save time for the team. 

When is the best time to test?

Just like a lot of things....it depends. As we have seen, from the previous sections, time is a critical factor that plays a role into how and why teams test. In the spirit of answering a question with a question...do you want to be FIRST to market? or do you want to be BEST in the market? With the help of automation, testing should be performed round the clock especially at the unit and integration testing levels.

The true value of testing has been masked by the notion that it is a process of finding bugs in a software application. A deeper introspection will reveal that it is a process of reducing the total cost of ownership for an application. When viewed from the contextual lens of unit testing, it will show that it is a means of documenting the business requirements in a living, breathing, version-able document. I have personally found unit tests to be the fastest way to understand what a software application is supposed to do. Change is a constant process in the life cycle of a software application. 

Unit tests provide peace of mind that existing functionality will not be broken during these changes. Writing a bug-free non-trivial application is virtually impossible, no matter the amount of testing performed. From a business value point of view, developers cannot spend a lifetime trying to deliver bug-free code. There is a compromise where business and IT have to decide what will be an acceptable level of testing and incorporate the necessary time needed to achieve this goal, no matter how much further it pushes the timeline for the delivery of the product. 

Taking it all back to my basketball analogy, every player (stakeholder) in a software application project needs to know how to wear their testing socks properly in order to avoid an unfortunate chain of events.