Evaluating the Vulnerability Detection Efficacy of Smart Contracts Analysis Tools

Smart contracts on modern blockchains pave the way to the development of novel application design paradigms, such as Distributed Applications (DApps). Interestingly, even some safety-critical systems are starting to adopt such a technology to devise new functionalities. However, being software, smart contracts are susceptible to flaws, posing a risk to the security of their users and thus making crucial the development of automatic tools able to spot such flaws. In this paper, we examine 11 real-world DApps that participated in security auditing contests on the Code4rena platform. We first conduct a manual analysis of the vulnerabilities reported during the contests and then assess whether state-of-the-art analysis tools can identify them. Our findings suggest that current tools are unable to reason on business logic flaws. Additionally, for other root causes, the detectors in these tools may be ineffective in some cases due to a lack of generality or accuracy. Overall, there is a significant gap between auditors’ findings and the results provided by these tools.