In the world of software testing, we are taught that all tests must pass in order for that test to be considered successful. A failed test step means a failed test, which generates a defect that has to be changed. That philosophy worked pretty well in Waterfall methodology projects where the entire application is developed before any testing occurs. However, it doesn’t work for Agile projects where features are added incrementally. In an Agile world, a failed test can actually be a successful test.
When It’s Okay to Fail
The point of regression testing is to ensure that an updated or new feature to an existing application doesn’t break the parts of the application that remained unchanged. As we have discussed previously, as an application grows, the regression test also grows to the point where it can’t be executed manually in the time allowed. So, we automate the regression test. But test scripts must change as the application changes. In an Agile world, there is barely enough time to test the changes, much less keep the automated regression test in sync.
What if we were to run the automated regression test, but predict which tests would fail and how they would fail? In this case, the expected result is a failed test step, so the result is a successful test – a successfully failed test.
Defining Success
In software testing it’s all about expected results. Usually the expected result is a passed test step, and any step that fails causes the entire test case to fail, prompting a defect to be written.
For example, changing a manual test script is easy — just change the document. You can even adjust the document early in the sprint while the feature is being developed. Changing an automated test script takes a little more time because you typically can’t make the change until the feature is released.
If you know where and how the automated regression test will fail though, you have essentially re-defined the expected results as a failed test. In this case, the test was successful because it failed exactly where you predicted. So, a failed test can be a successful test. During the next sprint cycle, change the automated test script to incorporate the new feature and you are back to expecting passed test steps.
Meeting Expectations
In software development, success is measured by systems that perform and operate as expected. Using failed test steps as success indicators requires a deeper understanding of the intricate interdependencies among the various features, modules, and interfaces that make up an application. It is this deeper understanding that allows you to predict failures, which are really successes.
For more information on how testing failures can lead to success, contact an expert at Stonemill Consulting today.
