A Test Case Selection Framework and Technique: Weighted Average Scoring Method

Authors

  • Rafaqat Kazmi Faculty of Computing, Universiti Teknologi Malaysia, 81310 Johor Bahru, Johor, Malaysia.
  • Dayang N. A. Jawawi Faculty of Computing, Universiti Teknologi Malaysia, 81310 Johor Bahru, Johor, Malaysia.
  • Radziah Mohamad Faculty of Computing, Universiti Teknologi Malaysia, 81310 Johor Bahru, Johor, Malaysia.
  • Imran Ghani School of Information Technology, Monash University Malaysia.
  • Muhammad Younas Faculty of Computing, Universiti Teknologi Malaysia, 81310 Johor Bahru, Johor, Malaysia.

Keywords:

Regression Testing, Test Case Selection, Test Suite Effectiveness,

Abstract

Test case selection techniques identify and eliminate the modification revealing test cases and try to reduce the test suite size for optimization of regression testing. The objective of this experiment is to assess the effectiveness of weighted average scoring method of test case selection against single objective test case selection techniques. The multiobjective test case selection with the weighted average scoring framework and technique are proposed in this study to select the test cases. This method is trying to solve conflicting test case selection objectives with six selection scenarios. The method used test data of cost, coverage, fault detection ability and code change information, convert them into the weighted average score as scalar function and presented this score to 100-index slabs related to low to high scores, then select the test cases. The results for these selection scenarios are computed and evaluated using size reduction of the test suite, inclusiveness, and precision. The results showed that all scenarios performed acceptable level within conditions applied from 17% minimum to 86% maximum in size reduction metrics. The inclusiveness showed 17% to 88% and 33% to 85% for precision metric.

References

J. B. Goodenough and S. L. Gerhart, “Toward a theory of test data selection,” IEEE Transactions on Software Engineering, vol. SE-1, no. 2, pp. 156-173, 1975.

K. F. Fischer, “A test case selection method for the validation of software maintenance modifications,” in Proceedings of COMPSAC, 1977, pp. 421-426.

B. Beizer, Black-Box Testing: Techniques for Functional Testing of Software and Systems. John Wiley & Sons, Inc., 1995.

M. Zavvar and F. Ramezani, “Measuring of software maintainability using adaptive fuzzy neural network,” International Journal of Modern Education & Computer Science, vol. 10, pp. 27-32, 2015.

M. A. Askarunisa, M. L. Shanmugapriya, and D. N. Ramaraj, “Cost and coverage metrics for measuring the effectiveness of test case prioritization techniques,” INFOCOMP Journal of Computer Science, vol. 9, no. 1, pp. 43-52, 2010.

S. Elbaum, P. Kallakuri, A. Malishevsky, G. Rothermel, and S. Kanduri, “Understanding the effects of changes on the cost‐ effectiveness of regression testing techniques,” Software Testing, Verification and Reliability, vol. 13, no. 2, pp. 65-83, 2003.

A. Orso, N. Shi, and M. J. Harrold, “Scaling regression testing to large software systems,” in ACM SIGSOFT Software Engineering Notes, 2004, pp. 241-251.

W. E. Lewis, Software Testing and Continuous Quality Improvement. CRC press, 2016.

W. E. Wong, J. R. Horgan, S. London, and H. A. Bellcore, “A study of effective regression testing in practice,” in ISSRE '97 Proceedings of the Eighth International Symposium on Software Reliability Engineering, 1997, pp. 264-274.

J. D. Musa, “Operational profiles in software-reliability engineering,” IEEE Software, vol. 10, no. 2, pp. 14-32, 1993.

C. Larman and V. R. Basili, “Iterative and incremental developments. a brief history,” Computer, vol. 36, no. 6, pp. 47-56, 2003.

A. Bertolino, “Software testing research: Achievements, challenges, dreams,” in Future of Software Engineering, 2007 (FOSE '07), 2007, pp. 85-103.

R. Kazmi, D. N. Jawawi, R. Mohamad, and I. Ghani, “Effective regression test case selection: a systematic literature review,” ACM Computing Surveys (CSUR), vol. 50, no.2, pp. 1-29, 2017.

R. Kazmi, I. Ghani, R. Mohamad, M. Tariq, I. S. Bajwa, and S. R. Jeong, “Trade-off between automated and manual testing: a production possibility curve cost model,” Int. J. Advance Soft Compu. Appl, vol. 8, no.1, pp. 12-27, 2016.

H. K. Leung and L. White, “A cost model to compare regression test strategies,” in Proceedings Conference on Software Maintenance, 1991, pp. 201-208.

T. Koju, S. Takada, and N. Doi, “Regression test selection based on intermediate code for virtual machines,” in Proceedings of International Conference on Software Maintenance (ICSM 2003), 2003, pp. 420-429.

G. Rothermel, M. J. Harrold, and J. Dedhia, “Regression test selection for C++ software,” Software Testing Verification and Reliability, vol. 10, pp. 77-109, 2000.

D. D. Nardo, N. Alshahwan, L. Briand, and Y. Labiche, “Coverage‐ based regression test case selection, minimization and prioritization: A case study on an industrial system,” Software Testing, Verification and Reliability, vol. 25, no. 4, pp. 371-396, 2015.

M. J. Harrold, J. A. Jones, T. Li, D. Liang, A. Orso, M. Pennings, S. Sinha, S. A. Spoon, and A. Gujarathi, “Regression test selection for Java software,” in OOPSLA '01 Proceedings of the 16th ACM SIGPLAN conference on Object-oriented programming, systems, languages, and applications 2001, pp. 312-326.

P. K. Chittimalli and M. J. Harrold, “Recomputing coverage information to assist regression testing,” IEEE Transactions on Software Engineering , vol. 35, no.4, pp. 452-469, 2009.

L. S. de Souza, R. B. Prudêncio, and F. d. A. Barros, “A Hybrid binary multi-objective particle swarm optimization with local search for test case selection,” in Intelligent Systems (BRACIS), 2014 Brazilian Conference, 2014, pp. 414-419.

A. A. L. de Oliveira, C. G. Camilo-Junior, and A. M. Vincenzi, “A coevolutionary algorithm to automatic test case selection and mutant in mutation testing,” in 2013 IEEE Congress on Evolutionary Computation (CEC), 2013, pp. 829-836.

E. D. Ekelund and E. Engström, “Efficient regression testing based on test history: An industrial evaluation,” in 2015 IEEE International Conference on Software Maintenance and Evolution (ICSME), 2015, pp. 449-457.

A. Konak, D. W. Coit, and A. E. Smith, “Multi-objective optimization using genetic algorithms: A tutorial,” Reliability Engineering & System Safety, vol. 91, no. 9, pp. 992-1007, 2006.

S. Wang, S. Ali, and A. Gotlieb, “Minimizing test suites in software product lines using weight-based genetic algorithms,” in Proceedings of the 15th Annual Conference on Genetic and Evolutionary Computation, 2013, pp. 1493-1500.

T. Hesterberg, “Weighted average importance sampling and defensive mixture distributions,” Technometrics, vol. 37, no. 2, pp. 185-194, 1995.

L. Inozemtseva and R. Holmes, “Coverage is not strongly correlated with test suite effectiveness,” in Proceedings of the 36th International Conference on Software Engineering, 2014, pp. 435-445.

P. K. Sen, “Estimates of the regression coefficient based on Kendall's Tau,” Journal of the American Statistical Association, vol. 63, no. 324, pp. 1379-1389, 1968.

Nayuki, Project Nayuki. University of Toronto, 2017. Available at https://www.nayuki.io/

JodaTime. Joda.org., 2017. Available at http://www.joda.org/jodatime/

Java Editor. The Eclipse Foundation. Available at https://eclipse.org/

M. R. Hoffmann, “Java Code Coverage,” 2017.

PIT. (2016). Available at http://pitest.org/

M. B. Doar, JDiff-An HTML Report of API Differences. LGPL. Available at http://javadiff.sourceforge.net/

Downloads

Published

2017-10-20

How to Cite

Kazmi, R., Jawawi, D. N. A., Mohamad, R., Ghani, I., & Younas, M. (2017). A Test Case Selection Framework and Technique: Weighted Average Scoring Method. Journal of Telecommunication, Electronic and Computer Engineering (JTEC), 9(3-4), 15–22. Retrieved from https://jtec.utem.edu.my/jtec/article/view/2912

Issue

Vol. 9 No. 3-4: Special Issue on Software Engineering II

Section

Articles

License

TRANSFER OF COPYRIGHT AGREEMENT

The manuscript is herewith submitted for publication in the Journal of Telecommunication, Electronic and Computer Engineering (JTEC). It has not been published before, and it is not under consideration for publication in any other journals. It contains no material that is scandalous, obscene, libelous or otherwise contrary to law. When the manuscript is accepted for publication, I, as the author, hereby agree to transfer to JTEC, all rights including those pertaining to electronic forms and transmissions, under existing copyright laws, except for the following, which the author(s) specifically retain(s):

  1. All proprietary right other than copyright, such as patent rights
  2. The right to make further copies of all or part of the published article for my use in classroom teaching
  3. The right to reuse all or part of this manuscript in a compilation of my own works or in a textbook of which I am the author; and
  4. The right to make copies of the published work for internal distribution within the institution that employs me

I agree that copies made under these circumstances will continue to carry the copyright notice that appears in the original published work. I agree to inform my co-authors, if any, of the above terms. I certify that I have obtained written permission for the use of text, tables, and/or illustrations from any copyrighted source(s), and I agree to supply such written permission(s) to JTEC upon request.