Guidelines for performing systematic literature reviews in software engineering

Recommended format for most reference management software

Recommended format for BibTeX-specific software

  • Kitchenham, B (Author)
  • Charters, S (Author)

guidelines for performing systematic literature reviews in software engineering kitchenham

The blue social bookmark and publication sharing system.

Log in with your username.

I've lost my password.

Log in with your OpenID-Provider.

  • Other OpenID-Provider

Guidelines for performing Systematic Literature Reviews in Software Engineering

  • engineering
  • evidence-based
  • research-methods
  • research.cs.softeng
  • research.support
  • systematic-literature-review
  • systematic-review

@jpmor

Comments and Reviews show / hide

Cite this publication, more citation styles.

BibSonomy is offered by the Data Science Chair of the University of Würzburg, the Information Processing and Analytics Group of the Humboldt-Unversität zu Berlin, the KDE Group of the University of Kassel, and the L3S Research Center .

Citations per year

Duplicate citations, merged citations, add co-authors co-authors, cited by view all.

Barbara  Ann Kitchenham

A systematic literature review on software security testing using metaheuristics

  • Published: 23 May 2024
  • Volume 31 , article number  44 , ( 2024 )

Cite this article

guidelines for performing systematic literature reviews in software engineering kitchenham

  • Fatma Ahsan 1 &
  • Faisal Anwer 1  

The security of an application is critical for its success, as breaches cause loss for organizations and individuals. Search-based software security testing (SBSST) is the field that utilizes metaheuristics to generate test cases for the software testing for some pre-specified security test adequacy criteria This paper conducts a systematic literature review to compare metaheuristics and fitness functions used in software security testing, exploring their distinctive capabilities and impact on vulnerability detection and code coverage. The aim is to provide insights for fortifying software systems against emerging threats in the rapidly evolving technological landscape. This paper examines how search-based algorithms have been explored in the context of code coverage and software security testing. Moreover, the study highlights different metaheuristics and fitness functions for security testing and code coverage. This paper follows the standard guidelines from Kitchenham to conduct SLR and obtained 122 primary studies related to SBSST after a multi-stage selection process. The papers were from different sources journals, conference proceedings, workshops, summits, and researchers’ webpages published between 2001 and 2022. The outcomes demonstrate that the main tackled vulnerabilities using metaheuristics are XSS, SQLI, program crash, and XMLI. The findings have suggested several areas for future research directions, including detecting server-side request forgery and security testing of third-party components. Moreover, new metaheuristics must also need to be explored to detect security vulnerabilities that are still unexplored or explored significantly less. Furthermore, metaheuristics can be combined with machine learning and reinforcement learning techniques for better results. Some metaheuristics can be designed by looking at the complexity of security testing and exploiting more fitness functions related to detecting different vulnerabilities.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price includes VAT (Russian Federation)

Instant access to the full article PDF.

Rent this article via DeepDyve

Institutional subscriptions

guidelines for performing systematic literature reviews in software engineering kitchenham

Abbreviations

Firefly algorithm

Cuckoo search

Genetic algorithm

Simulated annealing

Grammatical evolution

Genetic programming

Test object

Hill climbing

Memetic algorithm

Harmony search

Evolutionary programming

  • Evolutionary algorithm

Bat algorithm

Randomized algorithm

Evolutionary strategies

Differential evolution

Greedy search

Local Search

Null pointer exception

Cross site scripting

Standard genetic algorithm

Co-evolutionary algorithm

Hybrid genetic algorithm

Particle swarm optimization

Artificial bee colony optimization

Many independent objective

Hill climbing algorithm

Denial of service

Domain object model

Ant colony optimization

Improved genetic algorithm

Hill climbing using Korel’s AVM

K medoids algorithm

Hybrid evolutionary algorithm

Real-coded genetic algorithm

Whole test suite

Gene expression programming

Weighted genetic algorithm

Artificial bee colony algorithm

Memetic genetic algorithm

Structured query language injection

Extensible markup language injection

Multi-objective genetic algorithm

Dynamic principal component analysis

Multi-objective simulated annealing

Search-based software testing

Search-based software engineering

Common vulnerability scoring system

Co-operative co-evolutionary algorithm

Search-based software security testing

Multi-objective evolutionary search adaptive random testing

Fixed-sized candidate-set adaptive random testing

Collaborative co-evolutionary contract-driven algorithm

Multi-objective evolutionary algorithm based on decomposition

Multi-objective co-operative co-evolutionary algorithm

Evolutionary adaptive random testing algorithm

Dynamic multi-objective sorting algorithm

Non-dominated sorting genetic algorithm

Vector evaluated genetic algorithm

Niched pareto genetic algorithm

Afshan, S., McMinn, P., Stevenson, M.: Evolving readable string test inputs using a natural language model to reduce human oracle cost. In: 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation, pp. 352–361. IEEE (2013)

Afzal, W., Torkar, R., Feldt, R.: A systematic review of search-based testing for non-functional system properties. Inf. Softw. Technol. 51 (6), 957–976 (2009)

Article   Google Scholar  

Ahmed, M.A., Ali, F.: Multiple-path testing for cross site scripting using genetic algorithms. J. Syst. Architect. 64 , 50–62 (2016)

Ahsan, F., Anwer, F.: A critical review on search-based security testing of programs. Comput. Intell. Select Proc. InCITe 2022 , 207–225 (2023)

Almulla, H., Gay, G.: Learning how to search: generating effective test cases through adaptive fitness function selection. Empir. Softw. Eng. 27 (2), 1–62 (2022)

Alshahwan, N., Harman, M.: Automated web application testing using search based software engineering. In: 2011 26th IEEE/ACM International Conference on Automated Software Engineering (ASE 2011), pp. 3–12. IEEE (2011)

Alyasiri, H.: Evolving rules for detecting cross-site scripting attacks using genetic programming. In: International Conference on Advances in Cyber Security, pp. 642–656. Springer (2020)

Anand, S., Burke, E.K., Chen, T.Y., Clark, J., Cohen, M.B., Grieskamp, W., Harman, M., Harrold, M.J., McMinn, P., Bertolino, A., et al.: An orchestrated survey of methodologies for automated software test case generation. J. Syst. Softw. 86 (8), 1978–2001 (2013)

Anas, M., Imam, R., Anwer, F.: Elliptic curve cryptography in cloud security: a survey. In: 2022 12th International Conference on Cloud Computing, Data Science and Engineering (Confluence), pp. 112–117. IEEE (2022)

Andrews, A., Boukhris, S., Elakeili, S.: Fail-safe testing of web applications. In: 2014 23rd Australian Software Engineering Conference, pp. 200–209. IEEE (2014)

Anjum, M.S., Ryan, C.: Seeding grammars in grammatical evolution to improve search-based software testing. SN Comput. Sci. 2 (4), 1–19 (2021)

Anwer, F., Nazir, M., Mustafa, K.: Testing program for security using symbolic execution and exception injection. Indian J. Sci. Technol. 9 , 19 (2016)

Google Scholar  

Anwer, F., Nazir, M., Mustafa, K.: Safety and security framework for exception handling in concurrent programming. In: 2013 Third International Conference on Advances in Computing and Communications, pp. 308–311. IEEE (2013)

Anwer, F., Nazir, M., Mustafa, K.: Automatic testing of inconsistency caused by improper error handling: a safety and security perspective. In: Proceedings of the 2014 International Conference on Information and Communication Technology for Competitive Strategies, pp. 1–5 (2014)

Anwer, F., Nazir, M., Mustafa, K.: Security testing. Trends in Software Testing, pp. 35–66 (2017)

Anwer, F., Nazir, M., Mustafa, K.: Testing program crash based on search based testing and exception injection. In: International Conference on Security & Privacy, pp. 275–285. Springer (2019)

Arcuri, A.: Test suite generation with the many independent objective (MIO) algorithm. Inf. Softw. Technol. 104 , 195–206 (2018)

Arcuri, A.: Restful API automated test case generation with EvoMaster. ACM Trans. Softw. Eng. Methodol. 28 (1), 1–37 (2019)

Article   MathSciNet   Google Scholar  

Arcuri, A., Galeotti, J.P.: Handling SQL databases in automated system test generation. ACM Trans. Softw. Eng. Methodol. 29 (4), 1–31 (2020)

Arcuri, A., Galeotti, J.P.: Enhancing search-based testing with testability transformations for existing APIS. ACM Trans. Softw. Eng. Methodol. 31 (1), 1–34 (2021)

Arcuri, A.: Restful API automated test case generation. In: 2017 IEEE International Conference on Software Quality, Reliability and Security (QRS), pp. 9–20. IEEE (2017)

Arcuri, A.: Evomaster: Evolutionary multi-context automated system test generation. In: 2018 IEEE 11th International Conference on Software Testing, Verification and Validation (ICST), pp. 394–397. IEEE (2018a)

Avancini, A., Ceccato, M.: Comparison and integration of genetic algorithms and dynamic symbolic execution for security testing of cross-site scripting vulnerabilities. Inf. Softw. Technol. 55 (12), 2209–2222 (2013)

Avancini, A.: Security testing of web applications: a research plan. In: 2012 34th International Conference on Software Engineering (ICSE), pp. 1491–1494. IEEE (2012)

Avancini, A. and Ceccato, M.: Towards security testing with taint analysis and genetic algorithms. In:Proceedings of the 2010 ICSE Workshop on Software Engineering for Secure Systems, pp. 65–71 (2010)

Avancini, A., Ceccato, M.: Security testing of web applications: A search-based approach for cross-site scripting vulnerabilities. In: 2011 IEEE 11th International Working Conference on Source Code Analysis and Manipulation, pp. 85–94. IEEE (2011)

Avancini, A., Ceccato, M.: Grammar based oracle for security testing of web applications. In: 2012 7th International Workshop on Automation of Software Test (AST), pp. 15–21. IEEE (2012)

Aziz, B., Bader, M., Hippolyte, C.: Search-based sql injection attacks testing using genetic programming. In: European Conference on Genetic Programming, pp. 183–198. Springer (2016)

Balera, J.M., de Santiago Júnior, V.A.: A systematic mapping addressing hyper-heuristics within search-based software testing. Inf. Softw. Technol. 114 , 176–189 (2019)

Baluda, M.: Evose: evolutionary symbolic execution. In: Proceedings of the 6th International Workshop on Automating Test Case Design, Selection and Evaluation, pp. 16–19 (2015)

Baresel, A., Pohlheim, H., Sadeghipour, S.: Structural and functional sequence test of dynamic and state-based software with evolutionary algorithms. In: Genetic and Evolutionary Computation Conference, pp. 2428–2441. Springer (2003)

Baresel, A., Sthamer, H.: Evolutionary testing of flag conditions. In: Genetic and Evolutionary Computation Conference, pp. 2442–2454. Springer (2003)

Bejo, S. D., Assefa, B. G., Mohapatra, S. K.: Backip: Mutation based test data generation using hybrid approach. In: 2021 International Conference on Information and Communication Technology for Development for Africa (ICT4DA), pp. 178–183. IEEE (2021)

Benito-Parejo, M., Merayo, M. G.: Using genetic algorithms to select test cases for finite state machines with timeouts. In: 2021 IEEE Congress on Evolutionary Computation (CEC), pp. 2403–2410. IEEE (2021)

Bhattacharya, N., Sakti, A., Antoniol, G., Guéhéneuc, Y.-G., Pesant, G.: Divide-by-zero exception raising via branch coverage. In: International Symposium on Search Based Software Engineering, pp. 204–218. Springer (2011)

Boopathi, M., Sujatha, R., Kumar, C.S., Narasimman, S., Rajan, A.: Markov approach for quantifying the software code coverage using genetic algorithm in software testing. Int. J. Bio-Inspired Comput. 14 (1), 27–45 (2019)

Bottaci, L.: Instrumenting programs with flag variables for test data search by genetic algorithm. In: Proceedings of the 4th Annual Conference on Genetic and Evolutionary Computation, pp. 1337–1342 (2002)

CWE - Common Weakness Enumeration. https://cwe.mitre.org/

Cao, Y., Hu, C., Li, L.: An approach to generate software test data for a specific path automatically with genetic algorithm. In: 2009 8th International Conference on Reliability, Maintainability and Safety, pp. 888–892. IEEE (2009a)

Cao, Y., Hu, C., Li, L.: Search-based multi-paths test data generation for structure-oriented testing. In: Proceedings of the first ACM/SIGEVO Summit on Genetic and Evolutionary Computation, pp. 25–32 (2009b)

Castelein, J., Aniche, M., Soltani, M., Panichella, A., van Deursen, A.: Search-based test data generation for SQL queries. In: Proceedings of the 40th International Conference on Software Engineering, pp. 1220–1230 (2018)

Ceccato, M., Nguyen, C. D., Appelt, D., Briand, L. C.: Sofia: An automated security oracle for black-box testing of SQL-injection vulnerabilities. In: 2016 31st IEEE/ACM International Conference on Automated Software Engineering (ASE), pp. 167–177. IEEE (2016)

Chang, B.-M., Choi, K.: A review on exception analysis. Inf. Softw. Technol. 77 , 1–16 (2016)

Charmchi, M. R. H., Cami, B. R.: Paths-oriented test data generation using genetic algorithm. In: 2021 12th International Conference on Information and Knowledge Technology (IKT), pp. 157–162. IEEE (2021)

Costa, G., Valenza, A.: Why Charles can pen-test: an evolutionary approach to vulnerability testing (2020). arXiv preprint https://arxiv.org/abs/2011.13213

Cui, B., Liang, X., Wang, J.: The study on integer overflow vulnerability detection in binary executables based upon genetic algorithm. In: Foundations of Intelligent Systems, pp. 259–266. Springer (2011)

Dass, S., Namin, A. S.: Evolutionary algorithms for vulnerability coverage. In: 2020 IEEE 44th Annual Computers, Software, and Applications Conference (COMPSAC), pp. 1795–1801. IEEE (2020a)

Dass, S., Namin, A. S.: Vulnerability coverage as an adequacy testing criterion. arXiv preprint https://arxiv.org/abs/2006.08606 (2020b)

Dass, S., Namin, A. S.: Vulnerability coverage for adequacy security testing. In: Proceedings of the 35th Annual ACM Symposium on Applied Computing, pp. 540–543 (2020c)

Dass, S., Namin, A. S.: Vulnerability coverage for secure configuration (2020d). arXiv preprint https://arxiv.org/abs/2006.08604

de Almeida Biolchini, J.C., Mian, P.G., Natali, A.C.C., Conte, T.U., Travassos, G.H.: Scientific research ontology to support systematic review in software engineering. Adv. Eng. Inform. 21 (2), 133–151 (2007)

Del Grosso, C., Antoniol, G., Di Penta, M.: An evolutionary testing approach to detect buffer overflow. In: Student Paper Proceedings of the International Symposium of Software Reliability Engineering (ISSRE), St. Malo, France. Citeseer (2004)

Del Grosso, C., Antoniol, G., Di Penta, M., Galinier, P., Merlo, E.: Improving network applications security: a new heuristic to generate stress testing data. In: Proceedings of the 7th Annual Conference on Genetic and Evolutionary Computation, pp. 1037–1043 (2005)

de Lima, D. F., Albuquerque, D., Dantas Filho, E., Perkusich, M., Perkusich, A.: Integrating reinforcement learning in software testing automation: a promising approach. In: Anais do III Workshop Brasileiro de Engenharia de Software Inteligente, pp. 39–41. SBC (2023)

Duchene, F., Groz, R., Rawat, S., Richier, J.-L.: Xss vulnerability detection using model inference assisted evolutionary fuzzing. In:2012 IEEE Fifth International Conference on Software Testing, Verification and Validation, pp. 815–817. IEEE (2012)

Duchene, F., Rawat, S., Richier, J.-L., Groz, R.: Kameleonfuzz: evolutionary fuzzing for black-box XSS detection. In: Proceedings of the 4th ACM Conference on Data and Application Security and Privacy, pp. 37–48 (2014)

Eberlein, M., Noller, Y., Vogel, T., Grunske, L.: Evolutionary grammar-based fuzzing. In: International Symposium on Search Based Software Engineering, pp. 105–120. Springer (2020)

Ebert, F., Castor, F., Serebrenik, A.: An exploratory study on exception handling bugs in java programs. J. Syst. Softw. 106 , 82–101 (2015)

Elyasov, A., Prasetya, I. S., Hage, J.: Search-based test data generation for Javascript functions that interact with the dom. In:2018 IEEE 29th International Symposium on Software Reliability Engineering (ISSRE), pp. 88–99. IEEE (2018)

Esnaashari, M., Damia, A.H.: Automation of software test data generation using genetic algorithm and reinforcement learning. Expert Syst. Appl. 183 , 115446 (2021)

Fraser, G., Arcuri, A.: 1600 faults in 100 projects: automatically finding faults while achieving high coverage with EvoSuite. Empir. Softw. Eng. 20 (3), 611–639 (2015)

Fraser, G., Arcuri, A.: Evosuite: automatic test suite generation for object-oriented software. In: Proceedings of the 19th ACM SIGSOFT Symposium and the 13th European Conference on Foundations of Software Engineering, pp. 416–419 (2011)

Gan, J.-M., Ling, H.-Y., Leau, Y.-B.: A review on detection of cross-site scripting attacks (XSS) in web security. In: Advances in Cyber Security: Second International Conference, ACeS 2020, Penang, Malaysia, December 8–9, 2020, Revised Selected Papers 2, pp. 685–709. Springer (2021)

Gao, H., Feng, B., Zhu, L.: A kind of saaga hybrid meta-heuristic algorithm for the automatic test data generation. In: 2005 International Conference on Neural Networks and Brain, Vol. 1, pp. 111–114. IEEE (2005)

Del Grosso, C., Antoniol, G., Merlo, E., Galinier, P.: Detecting buffer overflow via automatic test input data generation. Comput. Oper. Res. 35 (10), 3125–3143 (2008)

Harman, M., Hu, L., Hierons, R. M., Baresel, A., Sthamer, H.: Improving evolutionary testing by flag removal. In: GECCO, pp. 1359–1366. Citeseer (2002)

Havrikov, N., Höschele, M., Galeotti, J. P., Zeller, A.: Xmlmate: Evolutionary xml test generation. In: Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering, pp. 719–722 (2014)

Htay, K. M., Othman, R. R., Amir, A., Zakaria, H. L., Ramli, N.: A pairwise t-way test suite generation strategy using gravitational search algorithm. In: 2021 International Conference on Artificial Intelligence and Computer Science Technology (ICAICST), pp. 7–12. IEEE (2021)

Hydara, I., Sultan, A.B.M., Zulzalil, H., Admodisastro, N.: Cross-site scripting detection based on an enhanced genetic algorithm. Indian J. Sci. Technol. 8 (30), 1–7 (2015)

Hydara, I., Sultan, A.B.M., Zulzalil, H., Admodisastro, N.: Current state of research on cross-site scripting (XSS)-a systematic literature review. Inf. Softw. Technol. 58 , 170–186 (2015)

Hydara, I., Sultan, A. B. M., Zulzalil, H., Admodisastro, N.: An approach for cross-site scripting detection and removal based on genetic algorithms. In: The Ninth International Conference on Software Engineering Advances ICSEA (2014)

Iannone, E., Di Nucci, D., Sabetta, A., De Lucia, A.: Toward automated exploit generation for known vulnerabilities in open-source libraries. In: 2021 IEEE/ACM 29th International Conference on Program Comprehension (ICPC), pp. 396–400. IEEE (2021)

Imam, R., Anwer, F., Nadeem, M.: An effective and enhanced RSA based public key encryption scheme (XRSA). Int. J. Inf. Technol. 14 (5), 2645–2656 (2022)

Imam, R., Anwer, F.: An empirical study of secure and complex variants of RSA scheme. In: Cyber Security, Privacy and Networking, pp. 185–196. Springer (2022)

Imam, R., Areeb, Q. M., Alturki, A., Anwer, F.: Systematic and critical review of RSA based public key cryptographic schemes: past and present status. IEEE Access (2021)

Imam, R., Kumar, K., Raza, S. M., Sadaf, R., Anwer, F., Fatima, N., Nadeem, M., Abbas, M., Rahman, O.: A systematic literature review of attribute based encryption in health services. J. King Saud Univ.-Comput. Inf. Sci. (2022b)

Jan, S., Panichella, A., Arcuri, A., Briand, L.: Automatic generation of tests to exploit xml injection vulnerabilities in web applications. IEEE Trans. Softw. Eng. 45 (4), 335–362 (2017)

Jan, S., Panichella, A., Arcuri, A., Briand, L.: Search-based multi-vulnerability testing of xml injections in web applications. Empir. Softw. Eng. 24 (6), 3696–3729 (2019)

Jan, S., Nguyen, C. D., Arcuri, A., Briand, L.: A search-based testing approach for xml injection vulnerabilities in web applications. In: 2017 IEEE International Conference on Software Testing, Verification and Validation (ICST), pp. 356–366. IEEE (2017a)

Jawed, M. S., Sajid, M.: Xecryptoga: a metaheuristic algorithm-based block cipher to enhance the security goals. Evolving Systems, pp. 1–22 (2022)

Kayacik, H. G., Heywood, M., Zincir-Heywood, N.: On evolving buffer overflow attacks using genetic programming. In: Proceedings of the 8th Annual Conference on Genetic and Evolutionary Computation, pp. 1667–1674 (2006)

Kayacik, H. G., Zincir-Heywood, A. N., Heywood, M.: Evolving successful stack overflow attacks for vulnerability testing. In: 21st Annual Computer Security Applications Conference (ACSAC’05), p. 8. IEEE (2005)

Khanna, M., Chauhan, N., Sharma, D., Toofani, A., Chaudhary, A.: Search for prioritized test cases in multi-objective environment during web application testing. Arab. J. Sci. Eng. 43 (8), 4179–4201 (2018)

Khari, M., Sinha, A., Verdu, E., Crespo, R.G.: Performance analysis of six meta-heuristic algorithms over automated test suite generation for path coverage-based optimization. Soft. Comput. 24 (12), 9143–9160 (2020)

Khari, M., Vaishali, Kumar, M.: Search-based secure software testing: a survey. In: Software Engineering: Proceedings of CSI 2015, pp. 375–381. Springer (2019)

Khor, S., Grogono, P.: Using a genetic algorithm and formal concept analysis to generate branch coverage test data automatically. In: Proceedings 19th International Conference on Automated Software Engineering, 2004, pp. 346–349. IEEE (2004)

Kitchenham, B., Charters, S.: Guidelines for performing systematic literature reviews in software engineering (2007)

Kumar, A., Nadeem, M., Banka, H.: Nature inspired optimization algorithms: a comprehensive overview. Evol. Syst., pp. 1–16 (2022)

Lin, Y., Ong, Y. S., Sun, J., Fraser, G., Dong, J. S.: Graph-based seed object synthesis for search-based unit testing. In: Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pp. 1068–1080 (2021)

Lin, Y., Sun, J., Fraser, G., Xiu, Z., Liu, T., Dong, J. S.: Recovering fitness gradients for interprocedural boolean flags in search-based testing. In: Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis, pp. 440–451 (2020)

Liu, G.-H., Wu, G., Tao, Z., Shuai, J.-M., Tang, Z.-C.: Vulnerability analysis for x86 executables using genetic algorithm and fuzzing. In: 2008 Third International Conference on Convergence and Hybrid Information Technology, vol. 2, pp. 491–497. IEEE (2008)

Liu, M., Li, K., Chen, T.: Security testing of web applications: a search-based approach for detecting SQL injection vulnerabilities. In: Proceedings of the Genetic and Evolutionary Computation Conference Companion, pp. 417–418 (2019)

Luo, Y.: Sqli-fuzzer: A SQL injection vulnerability discovery framework based on machine learning. In: 2021 IEEE 21st International Conference on Communication Technology (ICCT), pp. 846–851. IEEE (2021)

Lüdtke, S., Kraus, R., Barakat, R., Schneider, M. A.: Attack-based automation of security testing for IoT applications with genetic algorithms and fuzzing. In: 2021 IEEE 21st International Conference on Software Quality, Reliability and Security Companion (QRS-C), pp. 92–100. IEEE (2021)

Mann, M., Tomar, P., Sangwan, O.P.: Bio-inspired metaheuristics: evolving and prioritizing software test data. Appl. Intell. 48 (3), 687–702 (2018)

Mantere, T., Alander, J.T.: Evolutionary software engineering, a review. Appl. Soft Comput. 5 (3), 315–331 (2005)

Manès, V. J., Kim, S., Cha, S. K.: Ankou: guiding grey-box fuzzing towards combinatorial difference. In: Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering, pp. 1024–1036 (2020)

Mao, C.: Harmony search-based test data generation for branch coverage in software structural testing. Neural Comput. Appl. 25 (1), 199–216 (2014)

Mao, C., Wen, L., Chen, T. Y.: Adaptive random test case generation based on multi-objective evolutionary search. In: 2020 IEEE 19th International Conference on Trust, Security and Privacy in Computing and Communications (TrustCom), pp. 46–53. IEEE (2020)

Marashdeh, Z., Suwais, K., Alia, M.: A survey on SQL injection attack: Detection and challenges. In: 2021 International Conference on Information Technology (ICIT), pp. 957–962. IEEE (2021)

Marashdih, A. W., Zaaba, Z. F.: Detection and removing cross site scripting vulnerability in PHP web application. In:2017 International Conference on Promising Electronic Technologies (ICPET), pp. 26–31. IEEE (2017)

Marashdih, A. W., Zaaba, Z. F., Omer, H. K.: Web security: detection of cross site scripting in PHP web application using genetic algorithm. Int. J. Adv. Comput. Sci. Appl. (IJACSA) 8 (5) (2017)

Marculescu, B., Zhang, M., Arcuri, A.: On the faults found in rest APIs by automated test generation. ACM Trans. Softw. Eng. Methodol. 31 (3), 1–43 (2022)

McMinn, P.: Search-based software test data generation: a survey. Softw. Test. Verif. Reliab 14 (2), 105–156 (2004)

McMinn, P., Holcombe, M.: The state problem for evolutionary testing. In: Genetic and Evolutionary Computation Conference, pp. 2488–2498. Springer (2003)

McMinn, P., Shahbaz, M., Stevenson, M.: Search-based test input generation for string data types using the results of web queries. In: 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation, pp. 141–150. IEEE (2012)

Menéndez, H.D., Jahangirova, G., Sarro, F., Tonella, P., Clark, D.: Diversifying focused testing for unit testing. ACM Trans. Softw. Eng. Methodol. (TOSEM) 30 (4), 1–24 (2021)

Michael, C.C., McGraw, G., Schatz, M.A.: Generating software test data by evolution. IEEE Trans. Softw. Eng. 27 (12), 1085–1110 (2001)

Oster, N., Saglietti, F.: Automatic test data generation by multi-objective optimisation. In: International Conference on Computer Safety, Reliability, and Security, pp. 426–438. Springer (2006)

Padmanabhuni, B. M., Tan, H. B. K.: Light-weight rule-based test case generation for detecting buffer overflow vulnerabilities. In: 2015 IEEE/ACM 10th International Workshop on Automation of Software Test, pp. 48–52. IEEE (2015)

Paduraru, C., Melemciuc, M.-C., Stefanescu, A.: A distributed implementation using apache spark of a genetic algorithm applied to test data generation. In: Proceedings of the Genetic and Evolutionary Computation Conference Companion, pp. 1857–1863 (2017)

Panichella, A., Kifetew, F.M., Tonella, P.: Automated test case generation as a many-objective optimisation problem with dynamic selection of the targets. IEEE Trans. Software Eng. 44 (2), 122–158 (2017)

Panichella, A., Kifetew, F. M., Tonella, P.: Reformulating branch coverage as a many-objective optimization problem. In: 2015 IEEE 8th International Conference on Software Testing, Verification and Validation (ICST), pp. 1–10. IEEE (2015)

Pałka, D., Zachara, M., Wójcik, K.: Evolutionary scanner of web application vulnerabilities. In: International Conference on Computer Networks, pp. 384–396. Springer (2016)

Rauf, A., Anwar, S., Jaffer, M. A., Shahid, A. A.: Automated GUI test coverage analysis using GA. In: 2010 Seventh International Conference on Information Technology: New Generations, pp. 1057–1062. IEEE (2010)

Rawat, S., Ceara, D., Mounier, L., Potet, M.-L.: Combining static and dynamic analysis for vulnerability detection. arXiv preprint https://arxiv.org/abs/1305.3883 (2013)

Rawat, S., Mounier, L.: An evolutionary computing approach for hunting buffer overflow vulnerabilities: a case of aiming in dim light. In: 2010 European Conference on Computer Network Defense, pp. 37–45. IEEE (2010)

Ren, T., Wang, X., Li, Q., Wang, C., Dong, J., Guo, G.: Vulnerability mining technology based on genetic algorithm and model constraint. In: IOP Conference Series: Materials Science and Engineering, Vol. 750, p. 012168. IOP Publishing (2020)

Reungsinkonkarn, A., Apirukvorapinit, P.: Bug detection using particle swarm optimization with search space reduction. In: 2015 6th International Conference on Intelligent Systems, Modelling and Simulation, pp. 53–57. IEEE (2015)

Rodrigues, D.S., Delamaro, M.E., Corrêa, C.G., Nunes, F.L.: Using genetic algorithms in test data generation: a critical systematic mapping. ACM Comput. Surv. 51 (2), 1–23 (2018)

Romano, D., Di Penta, M., Antoniol, G.: An approach for search based testing of null pointer exceptions. In: 2011 Fourth IEEE International Conference on Software Testing, Verification and Validation, pp. 160–169. IEEE (2011)

Saber, T., Delavernhe, F., Papadakis, M., O’Neill, M., Ventresque, A.: A hybrid algorithm for multi-objective test case selection. In: 2018 IEEE Congress on Evolutionary Computation (CEC), pp. 1–8. IEEE (2018)

Seesing, A., Gross, H.-G.: A genetic programming approach to automated test generation for object-oriented software. Int. Trans. Syst. Sci. Appl. 1 (2) (2006)

Shahbazi, A., Miller, J.: Black-box string test case generation through a multi-objective optimization. IEEE Trans. Softw. Eng. 42 (4), 361–378 (2015)

Shuai, B., Li, H., Zhang, L., Zhang, Q., Tang, C.: Software vulnerability detection based on code coverage and test cost. In: 2015 11th International Conference on Computational Intelligence and Security (CIS), pp. 317–321. IEEE (2015a)

Shuai, B., Li, M., Li, H., Zhang, Q.: Test case generation for vulnerability detection using genetic algorithm. In: 4rd Int. Conf. Consumer Electronics, Communications and Networks, pp. 1198–1203 (2015)

Shuai, B., Li, M., Li, H., Zhang, Q., Tang, C.: Software vulnerability detection using genetic algorithm and dynamic taint analysis. In: 2013 3rd International Conference on Consumer Electronics, Communications and Networks, pp. 589–593. IEEE (2013)

Silva, R.A., de Souza, S. R. S., de Souza, P. S. L.: A systematic review on search based mutation testing. Inf. Softw. Technol. 81 , 19–35 (2017)

Skaruz, J., Seredynski, F.: Detecting web application attacks with use of gene expression programming. In: 2009 IEEE Congress on Evolutionary Computation, pp. 2029–2035. IEEE (2009)

Soltani, M., Derakhshanfar, P., Devroey, X., Van Deursen, A.: A benchmark-based evaluation of search-based crash reproduction. Empir. Softw. Eng. 25 , 96–138 (2020)

Sparks, S., Embleton, S., Cunningham, R., Zou, C.: Automated vulnerability analysis: leveraging control flow for evolutionary input crafting. In: Twenty-Third Annual Computer Security Applications Conference (ACSAC 2007), pp. 477–486. IEEE (2007)

Stallenberg, D. M., Panichella, A.: Jcomix: A search-based tool to detect xml injection vulnerabilities in web applications. In: Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pp. 1090–1094 (2019)

Thomé, J., Shar, L.K., Bianculli, D., Briand, L.: An integrated approach for effective injection vulnerability analysis of web applications through security slicing and hybrid constraint solving. IEEE Trans. Software Eng. 46 (2), 163–195 (2018)

Thomé, J., Gorla, A., Zeller, A.: Search-based security testing of web applications. In: Proceedings of the 7th International Workshop on Search-Based Software Testing, pp. 5–14 (2014)

Thomé, J., Shar, L. K., Bianculli, D., Briand, L.: Search-driven string constraint solving for vulnerability detection. In: 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE), pp. 198–208. IEEE (2017)

Tlili, M., Wappler, S., Sthamer, H.: Improving evolutionary real-time testing. In: Proceedings of the 8th Annual Conference on Genetic and Evolutionary Computation, pp. 1917–1924 (2006)

Tonella, P.: Evolutionary testing of classes. ACM SIGSOFT Softw. Eng. Notes 29 (4), 119–128 (2004)

Umar, K., Sultan, A. B., Zulzalil, H., Admodisastro, N., Abdullah, M. T.: Prevention of attack on Islamic websites by fixing SQL injection vulnerabilities using co-evolutionary search approach. In: The 5th International Conference on Information and Communication Technology for The Muslim World (ICT4M), pp. 1–6. IEEE (2014)

Umar, K., Sultan, A. B., Zulzalil, H., Admodisastro, N., Abdullah, M. T.: Formulation of SQL injection vulnerability detection as grammar reachability problem. In: 2018 International Conference on Information and Communication Technology for the Muslim World (ICT4M), pp. 179–184. IEEE (2018)

Vulnerability distribution of cve security vulnerabilities by types

Wang, W., Guo, X., Li, Z., Zhao, R.: Test case generation based on client-server of web applications by memetic algorithm. In: 2019 IEEE 30th International Symposium on Software Reliability Engineering (ISSRE), pp. 206–216. IEEE (2019a)

Wang, W., Wu, S., Li, Z., Zhao, R.: Parallel evolutionary test case generation for web applications. Inf. Softw. Technol. 155 , 107113 (2023)

Wang, Y., Wang, Y.: Use neural network to improve fault injection testing. In: 2017 IEEE International Conference on Software Quality, Reliability and Security Companion (QRS-C), pp. 377–384. IEEE (2017)

Wang, Y., Wu, Z., Wei, Q., Wang, Q.: Field-aware evolutionary fuzzing based on input specifications and vulnerability metrics. In: 2019 IEEE 10th International Conference on Software Engineering and Service Science (ICSESS), pp. 1–7. IEEE (2019b)

Wappler, S., Lammermann, F.: Using evolutionary algorithms for the unit testing of object-oriented software. In: Proceedings of the 7th Annual Conference on Genetic and Evolutionary Computation, pp. 1053–1060, (2005)

Wegener, J., Baresel, A., Sthamer, H.: Evolutionary test environment for automatic structural testing. Inf. Softw. Technol. 43 (14), 841–854 (2001)

Wegener, J., Buhr, K., Pohlheim, H.: Automatic test data generation for structural testing of embedded software systems by evolutionary testing. In: Proceedings of the 4th Annual Conference on Genetic and Evolutionary Computation, pp. 1233–1240 (2002)

Wei, Q., Li, Y., Zhang, Y.: A new method of evolutionary testing for path coverage. In: 2018 IEEE International Conference on Software Quality, Reliability and Security Companion (QRS-C), pp. 79–86. IEEE (2018)

Wu, Z., Atwood, J. W., Zhu, X.: A new fuzzing technique for software vulnerability mining. In: International Conference on Software Engineering. Citeseer (2009)

Xu, X., Jiao, L., Zhu, Z.: Boosting search based software testing by using ensemble methods. In: 2018 IEEE Congress on Evolutionary Computation (CEC), pp. 1–10. IEEE (2018)

Yao, X., Gong, D., Li, B., Dang, X., Zhang, G.: Testing method for software with randomness using genetic algorithm. IEEE Access 8 , 61999–62010 (2020)

Ye, J., Feng, C., Tang, C.: A fuzzer based on a fine-grained deeper strategy. In: 2017 4th International Conference on Information Science and Control Engineering (ICISCE), pp. 24–28. IEEE (2017)

Zhu, X. Y., Wu, Z. Y.: A new fuzzing technique using niche genetic algorithm. In: Advanced Materials Research, volume 756, pp. 4050–4058. Trans Tech Publ (2013)

Zhu, Z., Jiao, L., Xu, X.: Combining search-based testing and dynamic symbolic execution by evolvability metric. In: 2018 IEEE International Conference on Software Maintenance and Evolution (ICSME), pp. 59–68. IEEE (2018)

Download references

Author information

Authors and affiliations.

Department of Computer Science, Aligarh Muslim University, Aligarh, UP, 202002, India

Fatma Ahsan & Faisal Anwer

You can also search for this author in PubMed   Google Scholar

Contributions

All the authors are contributed equally.

Corresponding author

Correspondence to Fatma Ahsan .

Ethics declarations

Conflict of interest.

There is no Conflict of interest and no data available for this review paper.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Search string, selected primary studies, venue details and list of abbreviations, and quality assessment

See Tables 9 , 10 and 11 .

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Ahsan, F., Anwer, F. A systematic literature review on software security testing using metaheuristics. Autom Softw Eng 31 , 44 (2024). https://doi.org/10.1007/s10515-024-00433-0

Download citation

Received : 10 August 2023

Accepted : 13 March 2024

Published : 23 May 2024

DOI : https://doi.org/10.1007/s10515-024-00433-0

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

  • Meta-heuristic
  • Optimization algorithm
  • Software security testing
  • Code coverage
  • Program crash

Advertisement

  • Find a journal
  • Publish with us
  • Track your research

IMAGES

  1. Systematic literature review methodology based on Kitchenham (2004

    guidelines for performing systematic literature reviews in software engineering kitchenham

  2. Systematic Literature Review Stages (Adapted from Kitchenham (2004

    guidelines for performing systematic literature reviews in software engineering kitchenham

  3. Systematic Literature Review (Kitchenham n.d.)(Brereton et al. 2007

    guidelines for performing systematic literature reviews in software engineering kitchenham

  4. The phases of systematic review following the Kitchenham and Charters

    guidelines for performing systematic literature reviews in software engineering kitchenham

  5. (PDF) Kitchenham, B.: Guidelines for performing Systematic Literature

    guidelines for performing systematic literature reviews in software engineering kitchenham

  6. The systematic literature review process based on guidelines by

    guidelines for performing systematic literature reviews in software engineering kitchenham

VIDEO

  1. Powerful AI Techniques for Systematic Literature Reviews!

  2. Interpreting systematic literature reviews and Commonly used performance indicators in Social Protec

  3. Introduction to Systematic Review Software: Covidence

  4. Systematic Literature Review, Part 2: How

  5. Systematic Literature Review- Part 1, What and Why

  6. Systematic Literature Review: An Introduction [Urdu/Hindi]

COMMENTS

  1. Guidelines for performing Systematic Literature Reviews in Software Engineering

    The objective of this report is to propose comprehensive guidelines for systematic literature reviews appropriate for software engineering researchers, including PhD students. A systematic ...

  2. Guidelines for performing systematic literature reviews in software

    Guidelines for performing systematic literature reviews in software engineering. Print this record Download this record. RIS. Recommended format for most reference management software ... Guidelines for performing systematic literature reviews in software engineering. Technical report, EBSE Technical Report EBSE-2007-01. https://www.cs.auckland ...

  3. Systematic literature reviews in software engineering

    1.. IntroductionAt ICSE04, Kitchenham et al. [23] suggested software engineering researchers should adopt "Evidence-based Software Engineering" (EBSE). EBSE aims to apply an evidence-based approach to software engineering research and practice. The ICSE paper was followed-up by an article in IEEE Software [5] and a paper at Metrics05 [17].. Evidence-based research and practice was ...

  4. Guidelines for performing Systematic Literature Reviews in Software

    The objective of this report is to propose comprehensive guidelines for systematic literature reviews appropriate for software engineering researchers, including PhD students. A systematic literature review is a means of evaluating and interpreting all available research relevant to a particular research question, topic area, or phenomenon of ...

  5. Guidelines for performing Systematic Literature Reviews in Software

    Source: „Guidelines for performing Systematic Literature Reviews in SE", Kitchenham et al., 2007 Glossary 1/2 0.4 Executive Summary The objective of this report is to propose comprehensive guidelines for systematic literature reviews appropriate for software engineering researchers, including PhD students.

  6. PDF Procedures for Performing Systematic Reviews

    Procedures for Performing Systematic Reviews. Barbara Kitchenham. e-mail: [email protected]. Joint Technical Report. Software Engineering Group Department of Computer Science Keele University Keele, Staffs ST5 5BG, UK. Keele University Technical Report TR/SE-0401 ISSN:1353-7776. and.

  7. Systematic literature reviews in software engineering

    However, this does not appear to be a problem for systematic reviews in software engineering. For example, two recent meta-analyses reported fairly negative results but were still published ... B.A. Kitchenham, S. Charters, Guidelines for Performing Systematic Literature Reviews in Software Engineering Technical Report EBSE-2007-01, 2007. ...

  8. Systematic literature reviews in software engineering

    The recommended methodology for aggregating empirical studies is a systematic literature review (SLR) (see for example [4], [5], [6]). Kitchenham adapted the medical guidelines for SLRs to software engineering [7], and later updated them to include insights from sociology research [8]. SLRs are a means of aggregating knowledge about a software ...

  9. Performing systematic literature reviews in software engineering

    Context: Making best use of the growing number of empirical studies in Software Engineering, for making decisions and formulating research questions, requires the ability to construct an objective summary of available research evidence. Adopting a systematic approach to assessing and aggregating the outcomes from a set of empirical studies is also particularly important in Software Engineering ...

  10. ‪Barbara Ann Kitchenham‬

    Proceedings. 26th International Conference on Software Engineering, 273-281. , 2004. 1779 *. 2004. Systematic literature reviews in software engineering-a tertiary study. B Kitchenham, R Pretorius, D Budgen, OP Brereton, M Turner, M Niazi, ... Information and software technology 52 (8), 792-805.

  11. Systematic literature reviews in software engineering

    Systematic literature reviews in software engineering: A systematic literature review. Information and Software Technology , 51(1):7-15, 2009. Google Scholar; B. Kitchenham and S. Charters. Guidelines for performing systematic literature reviews in software engineering (version 2.3). Technical report, Keele University and University of Durham ...

  12. PDF Undertaking systematic reviews

    This document presents general guidelines for undertaking systematic reviews. The goal of this document is to introduce the methodology for performing rigorous reviews of current empirical evidence to the software engineering community. It is aimed primarily at software engineering researchers including PhD students.

  13. Systematic literature reviews in software engineering

    DOI: 10.1016/J.INFSOF.2008.09.009 Corpus ID: 3918101; Systematic literature reviews in software engineering - A systematic literature review @article{Kitchenham2009SystematicLR, title={Systematic literature reviews in software engineering - A systematic literature review}, author={Barbara Ann Kitchenham and Pearl Brereton and David Budgen and Mark Turner and John Bailey and Stephen G. Linkman ...

  14. Systematic Literature Reviews

    Kitchenham et al. report 53 unique systematic literature reviews in software engineering being published between 2004 and 2008 [103, 104]. They conclude that there is a growth of the number of systematic literature reviews being published, and that the quality of the reviews tend to be increasing too.

  15. Systematic review in software engineering

    Proceedings of the 26th International Conference on Software Engineering, (ICSE '04) Google Scholar Digital Library; Kitchenham, B.A. and S. Charters (2007) Guidelines for performing systematic literature reviews in software engineering, Technical Report EBSE-2007-01, School of Computer Science and Mathematics, Keele University. Google Scholar

  16. A systematic review of systematic review process research in software

    To that end we undertook a systematic review of papers that discuss problems with the current SR guidelines and/or propose methods to address those problems. Section 2 discusses the aims of our research, reports related research and identifies the specific research questions we address. Section 3 reports the search and paper selection process ...

  17. Systematic review in software engineering: where we are and where we

    This keynote will report the current results of an ongoing systematic literature review that aims to identify and categorise papers investigating the SLR process and the claims relating to that process. In 2004 Kitchenham et al. first proposed the idea of evidence-based software engineering (EBSE). EBSE requires a systematic and unbiased method of aggregating empirical studies and has ...

  18. Evidence-Based Software Engineering and Systematic Reviews

    The book is divided into three parts. The first part discusses the nature of evidence and the evidence-based practices centered on a systematic review, both in general and as applying to software engineering. The second part examines the different elements that provide inputs to a systematic review (usually considered as forming a secondary ...

  19. ‪Stuart Charters‬

    B Kitchenham, S Charters. Technical report, EBSE Technical Report EBSE-2007-01. , 2007. 12657 *. 2007. Does the technology acceptance model predict actual use? A systematic literature review. M Turner, B Kitchenham, P Brereton, S Charters, D Budgen. Information and software technology 52 (5), 463-479.

  20. Systematic literature reviews in software engineering

    This study assesses the impact of systematic literature reviews (SLRs) which are the ... This study has been undertaken as a systematic literature review based on the original guidelines as proposed by Kitchenham [22]. ... On the performance of hybrid search strategies for systematic literature reviews in software engineering. Information and ...

  21. Kitchenham Guidelines For Performing Systematic Literature Reviews in

    The document provides guidance on conducting a literature review according to the Kitchenham Guidelines for software engineering. It discusses the challenges of writing a literature review, including information overload, ensuring relevance, maintaining objectivity, and organizing information. The document then describes how a service called StudyHub.vip can assist with the literature review ...

  22. A systematic literature review on software security testing using

    The security of an application is critical for its success, as breaches cause loss for organizations and individuals. Search-based software security testing (SBSST) is the field that utilizes metaheuristics to generate test cases for the software testing for some pre-specified security test adequacy criteria This paper conducts a systematic literature review to compare metaheuristics and ...

  23. A systematic review of systematic review process research in software

    1. Introduction. In 2004 and 2005, Kitchenham, Dybå and Jørgensen proposed the adoption of evidence-based software engineering (EBSE) and the use of systematic reviews of the software engineering literature to support EBSE [18], [7].Since then, systematic reviews (SRs) have become increasingly popular in empirical software engineering as demonstrated by three tertiary studies reporting the ...