Software Testing Metrics

Go back to Tutorial

Software testing metrics, which are also known as software test measurement, indicates the extent, amount, dimension, capacity, as well as the rise of various attributes of a software process and tries to improve its effectiveness and efficiency imminently. Software testing metrics are the best way of measuring and monitoring the various testing activities performed by the team of testers during the software testing life cycle. Moreover, it helps convey the result of a prediction related to a combination of data. Hence, the various software testing metrics used by software engineers around the world are:

• Derivative Metrics: Derivative metrics help identify the various areas that have issues in the software testing process and allows the team to take effective steps that increase the accuracy of testing.
• Defect Density: Another important software testing metrics, defect density helps the team in determining the total number of defects found in a software during a specific period of time- operation or development. The results are then divided by the size of that particular module, which allows the team to decide whether the software is ready for the release or whether it requires more testing. The defect density of a software is counted per thousand lines of the code, which is also known as KLOC. The formula used for this is:

Defect Density = Defect Count/Size of the Release/Module

• Defect Leakage: An important metric that needs to be measured by the team of testers is defect leakage. Defect leakage is used by software testers to review the efficiency of the testing process before the product’s user acceptance testing (UAT). If any defects are left undetected by the team and are found by the user, it is known as defect leakage or bug leakage.

Defect Leakage = (Total Number of Defects Found in UAT/ Total Number of Defects Found Before UAT) x 100

• Defect Removal Efficiency: Defect removal efficiency (DRE) provides a measure of the development team’s ability to remove various defects from the software, prior to its release or implementation. Calculated during and across test phases, DRE is measured per test type and indicates the efficiency of the numerous defect removal methods adopted by the test team. Also, it is an indirect measurement of the quality as well as the performance of the software. Therefore, the formula for calculating Defect Removal Efficiency is:

DRE = Number of defects resolved by the development team/ (Total number of defects at the moment of measurement)

• Defect Category: This is a crucial type of metric evaluated during the process of the software development life cycle (SDLC). Defect category metric offers an insight into the different quality attributes of the software, such as its usability, performance, functionality, stability, reliability, and more. In short, the defect category is an attribute of the defects in relation to the quality attributes of the software product and is measured with the assistance of the following formula:

Defect Category = Defects belonging to a particular category/ Total number of defects.

• Defect Severity Index: It is the degree of impact a defect has on the development of an operation or a component of a software application being tested. Defect severity index (DSI) offers an insight into the quality of the product under test and helps gauge the quality of the test team’s efforts. Additionally, with the assistance of this metric, the team can evaluate the degree of negative impact on the quality as well as the performance of the software. Following formula is used to measure the defect severity index.

Defect Severity Index (DSI) = Sum of (Defect * Severity Level) / Total number of defects

• Review Efficiency: The review efficiency is a metric used to reduce the pre-delivery defects in the software. Review defects can be found in documents as well as in documents. By implementing this metric, one reduces the cost as well as efforts utilized in the process of rectifying or resolving errors. Moreover, it helps to decrease the probability of defect leakage in subsequent stages of testing and validates the test case effectiveness. The formula for calculating review efficiency is:

Review Efficiency (RE) = Total number of review defects / (Total number of review defects + Total number of testing defects) x 100

• Test Case Effectiveness: The objective of this metric is to know the efficiency of test cases that are executed by the team of testers during every testing phase. It helps in determining the quality of the test cases.

Test Case Effectiveness = (Number of defects detected / Number of test cases run) x 100

• Test Case Productivity: This metric is used to measure and calculate the number of test cases prepared by the team of testers and the efforts invested by them in the process. It is used to determine the test case design productivity and is used as an input for future measurement and estimation. This is usually measured with the assistance of the following formula:

Test Case Productivity = (Number of Test Cases / Efforts Spent for Test Case Preparation)

• Test Coverage: Test coverage is another important metric that defines the extent to which the software product’s complete functionality is covered. It indicates the completion of testing activities and can be used as criteria for concluding testing. It can be measured by implementing the following formula:

Test Coverage = Number of detected faults/number of predicted defects.

Requirement Coverage = (Number of requirements covered / Total number of requirements) x 100

• Test Design Coverage: Similar to test coverage, test design coverage measures the percentage of test cases coverage against the number of requirements. This metric helps evaluate the functional coverage of test case designed and improves the test coverage. This is mainly calculated by the team during the stage of test design and is measured in percentage. The formula used for test design coverage is:

Test Design Coverage = (Total number of requirements mapped to test cases / Total number of requirements) x 100

• Test Execution Coverage: It helps us get an idea about the total number of test cases executed as well as the number of test cases left pending. This metric determines the coverage of testing and is measured during test execution, with the assistance of the following formula:

Test Execution Coverage = (Total number of executed test cases or scripts / Total number of test cases or scripts planned to be executed) x 100

• Test Tracking & Efficiency: Test efficiency is an important component that needs to be evaluated thoroughly. It is a quality attribute of the testing team that is measured to ensure all testing activities are carried out in an efficient manner. The various metrics that assist in test tracking and efficiency are as follows:
• Passed Test Cases Coverage: It measures the percentage of passed test cases.

(Number of passed tests / Total number of tests executed) x 100

• Failed Test Case Coverage: It measures the percentage of all the failed test cases.

(Number of failed tests / Total number of test cases failed) x 100

• Test Cases Blocked: Determines the percentage of test cases blocked, during the software testing process.

(Number of blocked tests / Total number of tests executed) x 100

• Fixed Defects Percentage: With the assistance of this metric, the team is able to identify the percentage of defects fixed.

(Defect fixed / Total number of defects reported) x 100

• Accepted Defects Percentage: The focus here is to define the total number of defects accepted by the development team. These are also measured in percentage.

(Defects accepted as valid / Total defect reported) x 100

• Defects Rejected Percentage: Another important metric considered under test track and efficiency is the percentage of defects rejected by the development team.

(Number of defects rejected by the development team / total defects reported) x 100

• Defects Deferred Percentage: It determines the percentage of defects deferred by the team for future releases.

(Defects deferred for future releases / Total defects reported) x 100

• Critical Defects Percentage: Measures the percentage of critical defects in the software.

(Critical defects / Total defects reported) x 100

• Average Time Taken to Rectify Defects: With the assistance of this formula, the team members are able to determine the average time taken by the development and testing team to rectify the defects.

(Total time taken for bug fixes / Number of bugs)

• Test Effort Percentage: An important testing metric, test efforts percentage offer an evaluation of what was estimated before the commencement of the testing process vs the actual efforts invested by the team of testers. It helps in understanding any variances in the testing and is extremely helpful in estimating similar projects in the future. Similar to test efficiency, test efforts are also evaluated with the assistance of various metrics:
• Number of Test Run Per Time Period: Here, the team measures the number of tests executed in a particular time frame. (Number of test run / Total time)
• Test Design Efficiency: The objective of this metric is to evaluate the design efficiency of the executed test. (Number of test run / Total Time)
• Bug Find Rate: One of the most important metrics used during the test effort percentage is bug find rate. It measures the number of defects/bugs found by the team during the process of testing.

(Total number of defects / Total number of test hours)Number of Bugs Per Test: As suggested by the name, the focus here is to measure the number of defects found during every testing stage.

(Total number of defects / Total number of tests)

• Average Time to Test a Bug Fix: After evaluating the above metrics, the team finally identifies the time taken to test a bug fix.(Total time between defect fix & retest for all defects / Total number of defects)
• Test Effectiveness: A contrast to test efficiency, test effectiveness measures and evaluates the bugs and defect ability as well as the quality of a test set. It finds defects and isolates them from the software product and its deliverables. Moreover, the test effectiveness metrics offer the percentage of the difference between the total number of defects found by the software testing and the number of defects found in the software. This is mainly calculated with the assistance of the following formula:

Test Effectiveness (TEF) = (Total number of defects injected + Total number of defects found / Total number of defect escaped) x 100

• Test Economic Metrics: While testing the software product, various components contribute to the cost of testing, like people involved, resources, tools, and infrastructure. Hence, it is vital for the team to evaluate the estimated amount of testing, with the actual expenditure of money during the process of testing. This is achieved by evaluating the following aspects:
• Total allocated the cost of testing.
• The actual cost of testing.
• Variance from the estimated budget.
• Variance from the schedule.
• Cost per bug fix.
• The cost of not testing.
• Test Team Metrics: Finally, the test team metrics are defined by the team. This metric is used to understand if the work allocated to various test team members is distributed uniformly and to verify if any team member requires more information or clarification about the test process or the project. This metric is immensely helpful as it promotes knowledge transfer among team members and allows them to share necessary details regarding the project, without pointing or blaming an individual for certain irregularities and defects. Represented in the form of graphs and charts, this is fulfilled with the assistance of the following aspects:
  • Returned defects are distributed team member vise, along with other important details, like defects reported, accepted, and rejected.
  • The open defects are distributed to retest per test team member.
  • Test case allocated to each test team member.
  • The number of test cases executed by each test team member.\

 

Go back to Tutorial