As development teams look to be more responsive and agile in their processes, one of the key initiatives is to reduce waste. Identifying what waste to reduce is not always easy and sometimes we identify the wrong things to be removed. One of those common areas where we look to cut waste is in the use of decent requirements repository, test management or defect management tools. Sadly, these are the very things that they should actually be looking to hold onto as they are perhaps the biggest clues to efficiency that teams need. And the main reason why is traceability.
What is traceability and why is it so important?
Traceability is history and as any historian would teach you, the study of history is not just about the past, but an opportunity to see what you can learn from the past that will improve your future. Similarly, if we do not keep records of our past and where our errors are, we are likely to remake them in the future and not move forward as a development team. And repeating mistakes is the most inefficient way of developing software. Yet, we too easily fall into this trap anyway.
So what traceability are we looking for?
Well, it may vary from team to team, but if you cannot recall over the past few release cycles which user stories have caused the most issues, which tests failed the most, what their root cause is, and which functionality or type of functionality was affected the most – then you are likely to repeat all those same mistakes. And as cumbersome as management tools might appear, they are the best sources of providing this information.
So, while we need to improve our efficiencies and minimize waste across the board when it comes to clunky processes – we should not use this as an excuse for not following test best practices. If we are not keeping full traceability of our requirements, test, defects, and areas they impact, we are probably going to make those same mistakes repeatedly. And while you might think you are saving a lot of effort and time; the quality of your product and the satisfaction of your customers will suffer.
Never underestimate the value of a good process. There are many things we can do to improve the way we work with tools and reduce the efficiencies of the processes we have but don’t cut this one out entirely.
Below are some examples of the different types of traceability that you get and how they add value to your software project:
1. Forward Traceability
Forward traceability refers to the verification of requirements. This is where your requirements are linked to specific tests and measures (at code coverage, performance, or security scanning level) and tells the business that all tests relating to these requirements have been met and that this specific requirement is ready for release.
This is both helpful to the business as it makes it easy to track the development of a project but also vital to the team because it allows specific tests to be written to ensure all aspects of the requirements have been met and then have the developers design their code to go about meeting those requirements.
Forward traceability can also be used to see the impact that specific requirements or coding changes have on the testing impact, as teams can see the impact of these changes lower down and this helps them to better understand the risk of these changes and leads to smarter decision-making.
Forward traceability forms the basis for any test-driven development as well, as all the tests should be identified and scripted based on these requirements, though tests can also include a wider range of non-functional requirements that they need to adhere to as well.
2. Backward Traceability
Backward traceability refers to the linking of specific test failures and issues back to parts of the code and requirements, to help isolate where the issues may lie and speed up the fixing process.
This makes it easy for teams to gain specific context for a defect, which helps in identifying what needs to be fixed, provides them with more information for a root cause investigation, and better understand the impact that a defect has on the project. Vital information that can save teams critical time and again allow for smarter decisions to be made.
3. Prioritization
When we combine our different requirements and tests with priority levels it allows teams to make prioritized decisions on what tasks or work should be done first. Having clarity in what requirements, tests, or defects are more critical helps highlight what pieces of work should be prioritized by a team
Now that we have the benefits from this simple example in mind, we can talk about defining test traceability.
Test Traceability
When looking at traceability, it’s clear that while we have the forward and backward traceability that center from requirements to defects and back, those tests remain the key criteria for what should drive your traceability efforts.
We have already established that we can link specific tests to requirements and use that as both a guide for designing tests and tracing the coverage of our testing efforts against those requirements.
This test traceability provides the confidence to verify that the application works as expected. And confidence is not something that should be taken for granted. When a team is ready to say that code is ready to go live, you want to say that with a fair amount of confidence in the testing and quality effort leading up to that point. So, being able to confidently say that testing has met the criteria for go-live, allows teams to make these decisions easily.
And it is not just about the initial development, because as the application itself evolves, the tests themselves evolve to continually verify application functionality.
Why Test Traceability Matters?
Simply put, test traceability ensures that all functionality is verified. It enables you to quickly test specific areas of the application whenever you apply software fixes or add new functionality.
When your application is small, you know exactly what’s going on when things are broken. In addition, if your development team consists of one or two people, things are simple. When bugs appear, you simply fix them. However, the more complex your application becomes, and the more people become involved with the code, the more precise you must become to identify areas that need fixing.
Traceability helps manage project complexity
Traceability might not often seem like a big thing when we are thinking at a small level, but as teams and our software grows and becomes more complex, you will find it will add a lot of value to the development lifecycle of your system.
Keeping track of many different individual services, multiple application versions and working with the interdependencies within large systems become tricky and if you have the correct traceability in your testing it again helps the team to understand the impact of changes across all the different elements of these bigger systems ad can help to make sense of this complexity.
Traceability makes Parallel Development easier
I have mentioned working with multiple versions of software above, but it is not just about looking back to previous versions. In today’s development world a lot of development happens in parallel and there can often be differing versions in development at any given time. And you need traceability to help manage the complexity.
So, for example, while you might be working on a new feature that will form part of a version A of your product, there may be additional work on other projects that will also touch your particular services and form part of later B, C, and D versions – all with differing go-live dates, but all with developing overlapping at any given point.
In this scenario of parallel development, the team must know which login tests to run against which version of the application. When testing version A of the application, they should run the tests appropriate to version A. When testing version B or C of the application, the team must then ensure they’re running login tests appropriate for these versions.
This helps to ensure that you are testing smart as a team and helps to isolate any issues or defects that make only arise in certain versions of the code.
Traceability with Automated Testing Brings Confidence
Current CI/CD practices integrate and deploy new code very often. As a result, the system changes frequently. That means maintaining system stability while ensuring that the application meets business requirements becomes a delicate balancing act. Having automated testing in place ensures system stability and integrity as the system changes with development efforts.
Traceability allows us to have the confidence to know that our CI processes and respective tests are meeting the needs of the business, even through all the volatility of daily change and delivery.
While traceability should form part of all tests done on software systems, it’s important that we look at automated tests across different levels. And considering we normally work on our unit tests first; we need to ensure that we first build traceability not those lower-level tests before looking at higher-level tests.
Something which is often the opposite of what you see many teams doing as the testers focus on traceability only in their end-to-end and acceptance testers with developers not focusing on this at all, or very little. But if we want to make use of the full benefits of traceability in helping us build better software, we need to start traceability at a unit test level first before looking at higher-level tests.
Unit Tests
Before code is even written, teams should start scripting their unit tests and integrating them into the build process. Even when actual business requirements don’t go into detail on how specific modules should work, developers should understand what the code is supposed to do and how it plays its part in verifying these bigger requirements.
All these tests should then be linked directly through your tooling to those end requirements. We can either do this directly or break up our main requirements into smaller technical requirements for more granular tracking and better prioritization. However, doing this helps us to think through our development process better ad ensure that every part of the code we are writing serves its intended purpose with the desired outcomes clearly defined.
Once efficient unit tests cover most application code, confidence in system stability rises. Using modern tools, developers can understand how much of their code is covered with testing. This way they can trace unit tests.
Multiple-Layer Tests
The next layer of automatic testing creates automated tests that span several layers of the application. As compared to unit tests, multiple-layer tests span the API layer to the domain objects and storage layers. They test a slice of the application and thus cover the integration of the various layers of the application.
Teams can trace groups of requirements to these multiple layer tests as they span a lot of the codebase. At the same time, unit tests that form the foundation of these layered tests should also be linked, so that we can ensure our tests at the required layer when the lower layer tests are successful, but also to help identify issues when failures do occur.
User Interface Driven Tests
UI Driven tests may be the most complicated to automate, given the fragility of UI systems over procedural and reliable backend systems, slower performance, and the fragility of unmocked environments. However, from a traceability perspective, UI tests are often the easiest to link back to the original requirements, given that most analysis focuses on the user experience and therefore makes it easy to trace acceptance to a UI-driven test.
However, a common mistake is an overreliance on UI tests for acceptance criteria. While UI tests are easier for any analyst or product owner to go and validate themselves, you want to keep that separate from the actual criteria the technical development team uses and keep much of the focus of your tests at a lower, technical level, and leverage those as the basis for higher-level acceptance tests at a UI level.
CI Execution
I’ve spoken about linking your requirements, code, issues, and test altogether. However, our tests need to be executed several times a day – or at every build – and the best way to track all of this is in your CI tooling. Each fresh build or pipeline execution should be tracked with traceability showing which parts of the code and tests were executed in each (and thereby which requirements or issues may have been touched during this time).
Many CI jobs may not actually involve any change and the likelihood it brings different execution results to its underlying code is minimal. But it can help to pinpoint changes in other dependencies or environmental issues that occur at various times. History is again a vital aid here and correlating project issues to builds is a great way to better understand the deployment minutiae of your software system.
I do understand though that tracking execution for every CI job can be extremely taxing from a data storage perspective and so it’s important to clear out this data on a regular basis and rather use it to form trends that oy u can learn from unless you’re working through specific issues in which case you may want to store as much execution information as possible.
Database Layer
Another very common functional area that gets easily forgotten by many development teams is the database layer. With the focus on functional validation lying with the developed code, the data layer can easily be overlooked. And even if checking the DB forms part of the testing, you want to ensure that there is clear traceability between the code and the database areas that it covers.
This makes it easier for teams to measure test coverage at a DB level, better debug data-specific issues and perhaps test deeper at a data layer to cover a wider range of use cases. It can seem like an unnecessary bit of overhead to include specific SQL or DB information in your testing details, but it can add a lot of value in delivering a quality data layer in your system and reducing future maintenance efforts.
Invest in traceability tooling
Having all these layers of testing that cover your application code will require some tools that can help you maximize your confidence and keep track of your testing and requirements. And the more complex and the higher the number of systems in operation and development and the more you want to ensure there is clear traceability in place.
While some traceability tooling can be unnecessarily expensive, especially in how they want to integrate all the requirements and testing into their software and lock you in place, there are a lot of open source solutions that can help with this, with many CI tooling also catering for this to varying degrees. What matters more is not so much the tool itself, but rather ensuring that you cater for it within your testing frameworks itself and that a way to link across your different requirements, tests, CI pipeline execution, and issues exists.
Traceability is not just about linking information to testing and then storing the data for historical purposes. But also provides you and your team with critical information about the effectiveness of your development process and quality delivered across all the aforementioned layers. Using a variety of filters and dashboarding, if you have effective traceability tooling in place you can pinpoint specific issues and understand impacts clearly without needing to do an intensive investigation and the time it can save in helping to identify ways teams can improve is invaluable.
Test Tracing Proves Things Work
I know that having 100% test coverage for an application is a lofty ideal that often comes into conflict with the daily pressures of delivering software. However, having a clear testing strategy with an efficient way to trace application functionality to actual test cases will bring strong confidence in your product. In the end, test tracing proves applications work … or don’t when things fail along the way.
Traceability Matters
Making use of traceability in our projects, across all our layers of testing and linking them to our requirements and defects plays a vital role in understanding our systems better ad how we can design them better. So, even though traceability is often the last thing on many organizations' minds, it's vital to invest in tooling or frameworks that can help achieve this and build into the processes that allow you as a team to get the best out of your tracing.