Applications of Metamorphic Testing

Abstract

Software testing plays an important role in the software lifecycle. There are a set of test cases constructed based on certain requirements. Later, the software is tested based on these cases. A mechanism called “Oracle” is used to check whether a test is passed or failed in testing. There would be situations where this oracle would be difficult to apply, this where the “oracle problem” arises. The oracle more often is the human tester who does manual testing which may result in the code being less efficient and not verified. This paper mainly discusses on concepts, procedures of Metamorphic Testing.

Metamorphic testing helps to test the software without the use of an oracle which provides a solution to complex numerical problems, generate follow up cases automatically from the previously executed successful cases and detecting bugs through the follow up cases.

Introduction

Software reliability has always been an issue in Software Engineering. In order to resolve such issues during the development of the software, Software Testing was introduced. Testing examines the correctness of the software before they are released for production though it cannot prove the absence of code. Initially, testing included a lot of manual effort, as there was a progress in software, software testing also started update in its own ways. That’s when automation came into to play its role, which reduced the manual effort to some extent.

Testers used to follow a procedure called “oracle” to check whether the output of the program was successful or not. Oracle was used to compute if the expected output matches the observed output. There is certain situation where the output would be complex like complicated numerical simulations, predicting the correct output for such programs would be difficult and when compared with the observed output would be error prone. This raises a problem called the “Oracle Problem” and is considered to be one of the challenges in Software Testing.

When testing is done, certain set of test cases are listed based on the requirements. Even after these test cases are proved to be successful, there are chances that the software is still error prone.

In order to resolve the above issues quoted, an approach called “Metamorphic Testing” has been proposed by Chen et al. [2] to employ the successful test cases and alleviate the oracle problem. Based on the successful test cases, follow up cases can be generated by making references to metamorphic relations, that is, relations among multiple executions of the target program [9]. These tasks do not need an oracle. In this paper, I present the concepts of metamorphic testing and its effectiveness in solving the above raised issues.

Use of Metamorphic Testing (MT) in generating Follow up cases

In order to generate the follow up cases from the existing ones, metamorphic testing has to be combined with certain test strategies. Let’s consider a program implements a function on a domain along with certain test strategy. Based on the strategy, a test set is generated. If a program is run based on these test sets, a certain set of outputs is received. If an oracle is used, these outputs are verified against the observed output. Now the outputs are categorised into successful and failed. If it’s failed, the tester will debug the problem. The successful test cases will be documented and on these cases MT can be applied which helps in generating follow up cases. In this way the software is checked thoroughly.

MT generates the follow up cases by making references to “Metamorphic Relations”. MR is the property of its target function. MR has unique property to involve multiple executions. MT might check all the necessary conditions but may not be enough for software correctness. Because MT checks the relations among several executions rather than the correctness of individual outputs, MT does not need an oracle and can be fully automated.

Role of MT in Detecting a Bug In Follow Up Cases

MT has been proved as better technique when compared to other checking techniques. MT includes multiple executions of the program. Inputs of these multiple executions and their outputs are expected to satisfy certain properties of the algorithm. Those properties are called “Metamorphic Relations”. MR can be associated with the source cases to generate the follow up cases. Once the follow up cases are generated, execute both the source and follow up cases to check their relations. If the output of both the cases violate the MR they are said to have failed and through this we get an indication that the program under test contains a bug. Consider an example where the source case searches for a word “technique” resulting in 1M results, whereas the follow up case searches for the word “testing technique” resulting in 4564k results, in such a situation the Metamorphic relation is said to have been satisfied. If the results of the follow up case are more than the source case, then MR is said to have violated which results in a failure, indicating a bug in the software.

Metamorphic Testing and the Oracle Problem

When working with numerical programs, testers provide simple inputs in order to get the required output. When simple inputs are provided, it doesn’t enhance the confidence towards the correctness of the software. This result in an issue called “oracle problem”. Metamorphic Testing came into use in order to alleviate this problem. Metamorphic Relations play an important role in such a situation. Quoting an example from [3], where a tester tries to compute the shortest path between a source vertex and a destination vertex, follow up cases are generated from the sources case. Both are executed and based on the results the MR test is said to be passed or failed.

MT is not just restricted to numerical problems; it can be used for any kind of problems like graphs, compiler, computer graphics etc. To answer the question if MT can be conducted by testers, Yes they can perform. Based on the Empirical Studies [9] it’s proven that testers can write metamorphic relations and conduct the required tests.

Conclusion

This article introduced the concept of MT and how it played a role in answering few research questions. The Oracle problem had always been an issue for testers. When MT was introduced, it ease the approach of solving such issues. Many industries are not aware of this approach and it’s benefits. The unique character of MT is that it doesn’t require a human to generate the follow up cases. Empirical studies [7][9] have been conducted to conclude that MT helps to alleviate the oracle problem. As Metamorphic Relations exists in both numerical and non-numerical areas, MT can be considered as a practical approach and applied to Real life scenarios. These various researches prove that MT is gaining maturity and would be very effective in the future of Testing.

Metamorphic Testing has always been a thriving topic and can be applied on multiple domains. Although Metamorphic Relation identified through adhoc manner have good performance, if it is identified through a systematic manner, the performance would be even better. Future research can be conducted on choosing which effective metamorphic relations can be used as MR is the main reference in Metamorphic Testing.

REFERENCES

Morell, L.J. A theory of fault-based testing, IEEE Transactions on Software Engineering, 16(8), 1990, pp. 844– 857

Chen, T. Y., Cheung, S. C., and Yiu, S. M. Metamorphic testing: a new approach for generating next test cases, Technical Report HKUST-CS98-01, Department of Computer Science, Hong Kong University of Science and Technology, Hong Kong, 1998.

Gotlieb, A. and Botella, B. Automated metamorphic testing, In Proceedings of the 27th Annual International Computer Software and Applications Conference (COMPSAC 2003), IEEE Computer Society Press, Los Alamitos, California, 2003, pp. 34–40.

T. Chen, D. Huang, H. Huang, T.-H. Tse, Z. Yang, Z. Zhou, "Metamorphic testing and its applications", Proc. 8th Int. Symp. Future Softw. Technol., pp. 310-319, 2004.

C. Murphy, K. Shen, and G.E. Kaiser, “Automatic System Testing of Programs without Test Oracles,” Proc. 18th Int’l Symp. Software Testing and Analysis (ISSTA ’09), pp. 189-200, 2009.

Yao, Y, Huang, S, Ji, M. (2012). Research on metamorphic testing for oracle problem of integer bugs. In Jin D, & Lin, S (Eds.) Advances in computer science and information engineering (pp. 95–100). Berlin: Springer.

H. Liu, F.-C. Kuo, D. Towey, and T. Y. Chen, “How effectively does metamorphic testing alleviate the oracle problem?” Software E ngineering,IEEETransactionson,vol.40,no.1,pp.4–22,Jan2014.

T. Y. Chen, “Metamorphic testing: A simple method for alleviating the test oracle problem,” in Proceedings of the 10th International Workshop on Automation of Software Test, ser. AST ’15. Piscataway, NJ, USA: IEEE Press, 2015, pp. 53–54. [Online]. Available: http://dl.acm.org/citation.cfm?id=2819261.2819278

Sergio Segura, Gordon Fraser, Ana B. Sánchez, and Antonio Ruiz-Cortés. 2016. A survey on metamorphic testing. IEEE Transactions on Software Engineering 42, 9, 805--824.