Researchers from the URV and the University of Piraeus design a system to detect malware that modifies its behaviour to avoid detection

A study co-led by the Universitat Rovira i Virgili has developed a tool capable of identifying sophisticated malicious software —also known as malware—  that avoids the usual detection techniques. The model analyses all the possible ways in which malware could be executed, even when it modifies its usual behaviour to avoid detection. The results show that this new system is more efficient than current commercial tools, with an accuracy of about 99% in the detection of malicious files.

As technology evolves, computer attacks are becoming increasingly sophisticated. Many digital threats manage to penetrate traditional defences with the aim of stealing personal data, sabotaging critical infrastructures —such as the health or energy sectors— or committing financial fraud. Behind the attacks there are cybercriminals seeking financial gain or organised groups of hackers who exploit vulnerabilities to destabilise companies, institutions or governments.

One of the most commonly used methods for detecting malware is to run it in a virtual environment or sandbox. These simulators imitate a real operating system to observe the behaviour of the virus. Thus, if they detect that a suspected file is behaving in a harmful way, they can quarantine or delete it. However, many malicious programs have evolved to recognise these environments and hide their activity to avoid detection.

Symbolic execution

To address this challenge, the researchers have implemented a model that incorporates symbolic execution. Unlike traditional systems, which can only see how the virus acts in a specific scenario, this technique makes it possible to analyse all possible situations in which the malicious code could be executed, even anticipating hidden behaviour. Fran Casino, a researcher at the Department of Computer Engineering and Mathematics, explains that to test the new system, they used a reference database with more than 14,000 malware samples and more than 1,500 harmless files —such as legitimate applications of the operating system that do not represent any risk to users.

Furthermore, in order to optimise the process, a classification method was used to group similar files together. “This allowed us to analyse only those files that did not have a known structure, thus reducing the workload without compromising the accuracy of the system” Casino recalls.

Superior detective skills

The model demonstrated a superior ability to detect malware in a variety of operating systems compared to commercial sandbox-type tools. More specifically, it was able to correctly identify about 99% of malware samples, and thus outperformed commercial alternatives, which only accurately detected 78% of the same samples.

The symbolic execution system also outperformed the sandbox environments in detecting false positives. It managed to correctly classify 93% of the harmless files, while the reference sandbox models only managed 89%. This is a notable improvement as it reduces the possibility of legitimate programs being blocked by mistake.

Another of the outstanding advantages of the symbolic execution method is its efficiency. While traditional systems require between thirty seconds and four minutes to analyse each file, the new system takes on average just over 30 seconds to do so. This represents a significant saving of time and resources and makes a difference in situations where large quantities of files have to be analysed every day. Beyond that, the new system provides detailed information on the inner workings of malicious code that can prove useful for improving protection measures.

Despite the potential of this new model, Casino points out that there are still challenges to be overcome. “One of the main challenges is to use advanced dynamic libraries to adapt symbolic execution to complex software and programming,” he says. Moreover, although symbolic execution is a great option for exhaustive sample analysis, existing models still need to be fully developed to be viable solutions in a conventional operating system.

Reference: Vasilis Vouvoutsis, Fran Casino, Constantinos Patsakis. Beyond the sandbox: Leveraging symbolic execution for evasive malware classification, Computers & Security, Volume 149, 2025, 104193. ISSN 0167-4048, https://doi.org/10.1016/j.cose.2024.104193