Test Coverage

Go back to Tutorial

Test coverage is defined as a metric in Software Testing that measures the amount of testing performed by a set of test. It will include gathering information about which parts of a program are actually executed when running the test suite in order to determine which branches of conditional statements have been taken.

In simple terms, it is a technique to ensure that your tests are actually testing your code or how much of your code you exercised by running the test.

Test coverage and code quality are two of a handful of fundamental metrics used to analyse, track and measure the effectiveness of an IT project or initiative.

Both test coverage and code quality are interlinked in a way few other metrics are. For instance, one of the ways we measure code quality is by looking at corresponding test coverage.

Test Coverage aims to measure of the effectiveness of testing in a qualitative manner. It determines whether the test cases are covering entire functional requirements. You can think of it as a kind of black box testing, where test cases are not written based on code but based on user requirements or expected functionality.

The common mechanisms used for test coverage measurement are unit testing, functional testing, performance testing, integration or system testing and acceptance testing.

Unit tests are written at a granular level to check if a function/method performs as expected. In functional testing, each functionality mentioned in the requirement document is tested. Performance testing is generally a way to stress test the code for its responsiveness and stability at different workloads. Integration testing or system testing is done to test if completely integrated product works in expected manner. Acceptance testing is generally done at the end of the development cycle. For acceptance testing, the product is handed over to the stakeholders and tested by them to determine whether the product is in acceptable state.

Tools

These tools aim to measure the testing effort in a qualitative manner. The idea here is to see if all requirements and functionalities are logically covered via testing. So, this is not straight-forward to measure as compared to code coverage. Methodologies like TDD (Test Driven Development) are also helpful to analyse test coverage and bring in the discipline to add tests from get go. To measure the impact of such tests, you will need to manually list out requirements and then analyse test case to understand which of those are covered.

Still, you need frameworks to write tests. Let’s see some testing frameworks available for various programming languages.

JUnit, the well known Java testing framework serves as a foundation to launch tests on the Java Virtual Machine. With support for assertions, assumptions, default test methods and other mechanisms to test functionality JUnit is a very good choice for testing Java based software.

For Python, framework called PyUnit is available. As the name indicates it is the JUnit equivalent for Python. But there are several other testing frameworks available to help you as well.

Go also offers a built in testing package for programs letting you easily write, organize and run tests. It also offers various profiling tools to understand software behaviour under varying loads.

Test Coverage

• Finds the area of a requirement not implemented by a set of test cases
• Helps to create additional test cases to increase coverage
• Identify a quantitative measure of test coverage, which is an indirect method for quality check
• Identify meaningless test cases that do not increase coverage

Applying Test Coverage Metric

• Test coverage can be done by exercising the static review techniques like peer reviews, inspections, and walkthrough
• By transforming the ad-hoc defects into executable test cases
• At code level or unit test level, test coverage can be achieved by availing the automated code coverage or unit test coverage tools
• Functional test coverage can be done with the help of proper test management tools

Benefits

• As a black-box testing methodology this approach is aligned to the requirement specifications and has minimal interaction with code itself. Isolating tests from code allows skew-free tests.
• This tests the software features, making it a good candidate for other criteria like acceptance tests.

Disadvantages

• Most of the tasks in such testing methodology are manual. Since there are no tools to automate, it takes lot of effort to analyze the requirements and create test cases.
• There is no concrete way to measure such coverage, you can of course count features and then measure against number of tests, but that still leaves space for judgement errors. On the other hand code coverage is much more concretely measured.

Test coverage and LOC (Lines of code)

You simply take:

(A) the total lines of code in the piece of software you are testing, and

(B) the number of lines of code all test cases currently execute, and

Find (B divided by A) multiplied by 100 – this will be your test coverage %.

For example,

If the total lines of code in a system component is 1000 and the number of lines being actually executed through all existing test cases is 650, then your test coverage is:

(650 / 1000) * 100 = 65%

What is the generally accepted ‘sufficient’ test coverage when measured by number of lines of code executed? The consensus hovers around 80% – higher for critical systems (definition of critical may vary by industry, geography, user base etc.).

The important question is: Does this metric work?

Generally, people think 80% is adequate. Then again, there is the question of whether you’re using good tests rather than those that aren’t really useful for coverage.

TDD and Test coverage

Let’s consider how to run  Agile project (week-long sprints as an example).

Sprint – Day 1 – My scrum team use the planning session to shortlist the top x stories on the backlog for delivery. Each of the stories has been elaborated and story pointed in readiness for the planning sessions (through ongoing backlog grooming sessions). A Scrum Tester (or testers) picks up the freshly minted sprint backlog to map out all the test scenarios they expect to cover with their test cases. These scenarios are then handed over to the Developers who are already plugging away at code.

Sprint – Day 2 – By this time, scrum developers have made some initial headway with dev planning and kick off, and have had a chance to get an overview of the requirements and test scenarios for the sprint with the bas and testers. The testers have completed scripting new test cases or copying over existing test cases from a case repository as applicable, to cover off all the test scenarios. They’ve also identified both Happy and Unhappy flows, and prioritised these. And finally, they’ve mapped the test cases back to the source requirement, establishing traceability. By this point, the testers know if all the requirements for the current sprint have been covered by test cases.

Sprint – Final Day – Scrum Developers have used the test scenarios and detailed test cases to guide their effort at coding, and have unit tested the code with these test cases as well. By the time they deploy to an Integrated Test Environment for end-to-end functional testing, most of the easy to find, unseemly bugs have been found. Scrum Testers have also begun to execute end-to-end tests using these test cases, and help identify the high/critical bugs that prevent a story from being delivered in that sprint. When we get to the Demo and Retrospective, usually all high and critical bugs will have been fixed, with other unresolved bugs being prioritised for future resolution as necessary (We always end a release by mopping up most of these ‘other’ bugs with a spike).

As you can see, by the time we get to the end of a Sprint, or a Release, my team have a way of ensuring all requirements are covered adequately by tests, and that we meet Exit Criteria from Testing to Release (e.g.: No High/Critical bugs, 100% test case execution, 95% tests passed).

With Test-Driven Development, this is truly achievable. And if you are able to demo a working, almost-bug-free product every Sprint, and follow this up with mop-up Spikes to fix the remaining unseemly bugs, my experience suggests you’re in a good place to go to Release.

Beyond Test Coverage

Coverage is an important but limited metric. Verifying testing breadth in the traditional sense is a useful concept, but one that is difficult to measure operationally. Most teams do not have a clear measure of the actual comprehensiveness of their testing efforts.

What would be truly useful is to measure actual test scope, going beyond unit tests to include integration tests, acceptance tests, and manual tests as well. Traditionally there has been no easy way to see a unified test coverage metric across all types of tests and all test systems in one place.

Go back to Tutorial