Exploring the effectiveness of chatGPT in Smart Contract Auditing
It was 30th November, 2022, when ChatGPT was launched. It did not take it long to take the world by storm. No matter what social media you use, there are posts, memes, informative articles and whatnot on chatGPT. Not only that, the chatGPT was the talk of the mainstream media. There are no second thoughts when I say everyone talked about chatGPT and its power.
In this blog, let’s discuss how chatGPT is used or can be used in smart contract auditing or Web3 cyber security. Let’s first start with what chatGPT exactly is.
What is ChatGPT?
ChatGPT is an interactive chatbot that takes in prompts and returns answers based on its trained data. It has a remarkable ability to communicate in conversational dialogue and provide responses that can seem surprisingly human.
Apart from that, one of the things that make it smarter is its unique ability to keep on learning from the user’s input data; this is implemented in a layer of Reinforcement learning with human feedback (RLHF), which helps it return answers that are satisfying to humans.
Training data
Every AI model is nothing but a trained machine that gives answers based on its learning and findings from the training data. The training data can be anything from videos to text which is fed to a model which learns about this data, and when a problem is proposed to this model, based on its learning from the training data, it gives answers.
The chatGPT was trained on the data collected from the internet, including sources such as Reddit discussions, to help ChatGPT learn dialogue and achieve a human-like response style. chatGPT is also trained on human feedback. This technique is called Reinforcement Learning with Human Feedback so that the AI learns what people expect when they ask a question.
ChatGPT can find Vulnerabilities
Long after its release, people started experimenting with the capabilities of chatGPT in various use cases and scenarios. This experimentation was also done in smart contract security.
And chatGPT sure did not fail us. However, it still has room for improvement, but it proved to be useful and of significant help to auditors and people dealing with smart contracts. When it comes to the well-known hacks and some hacks which have been in the system for quite some time now, it is very useful in catching them.
Some of the common vulnerabilities that chatGPT finds with a bit of accuracy are:-
- Renterancy attack: This is a common vulnerability in which an attacker can repeatedly call a function within a smart contract before the previous execution has been completed, leading to unexpected or malicious behaviour.
- Integer Overflows/Underflows: Smart contracts often rely on integer calculations, and if these calculations are not properly checked, they can result in unexpected or incorrect behaviour.
- Unchecked return values: A contract may not properly handle unexpected return values from external calls, which may lead to a potential vulnerability and can cause harm.
- Unprotected functions: A contract may not have proper access control, leading to unauthorised access to sensitive functions. Which can lead to heavy loss.
There are some other vulnerabilities and issues that chatGPT can identify with smart contracts, and you will surely be surprised to see them. Still, through our tests, we discovered that you would often receive a false alarm, and there is a huge possibility that some crucial bugs are missed.
Can chatGPT find all vulnerabilities?
While chatGPT is a useful tool and a breakthrough of AI for the masses, it is still far from perfect and cannot be left to completely secure smart contracts.
Our test found that the chatGPT raised a false alarm for a re-entrance attack, which was already guarded and tested. Apart from that, there were some more false alarms, and most importantly, the critical bug our team found was completely ignored by chatGPT. Let’s discuss some of the things chatGPT is likely to miss.
- Project-specific Logic:- The project’s backbone is its logic and how things are interconnected, but chatGPT seems to miss on it. During tests, it was found that chatGPT was often unable to find the critical bug, which was logic specific. Due to the complexity of the underlying infrastructure of the protocol, chatGPT misses the critical vulnerabilities that arise due to the interconnection of contracts to fulfil the logical requirement of the project.
- Inaccurate math calculation and statistical models:- When it comes to projects, whether it’s a gaming project, a DeFi project or can be anything, it mostly involves mathematical calculations and relations. These formulae are often left unchecked and unmonitored by the chatGPT, and potential bugs are missed.
- Irregularities in Intended design and Implementation:- Many times, the implementation by the developers is not as correct as it should be, leading to security issues. This has been exploited in the past and continues to be one of the essential sectors that can be improved, and chatGPT is a bit ignorant on this front as well.
Why can’t chatGPT do it all?
No doubt, it can be of great aid to the developers. Still, it can only partially be trusted with smart contract auditing as it clearly needs to cover up all the vulnerabilities and leave some of the crucial ones unchecked, which can be devastating. In this section, let’s discuss why chatGPT is not ready yet.
The chatGPT needs to reason well about some of the higher-level topics. It may produce inconsistent results or provide incomplete solutions in such cases. Ownership of contracts, re-entrance, and fee distribution are some of the higher-level topics that chatGPT needs to grasp most of the time.
Integrating the Large Language Models(LLM) with the software is something which needs to be worked upon. New advancements are trying to improve in this field, but we are still looking for a breakthrough. This field is relatively new, so we will see a good investment to improve LLM’s performance. Different tools need to create, which will help developers interact better.
Prompt creation is another thing that developers need to find their way through sometimes. Prompts are really crucial when working with the LLMs. The prompt dictates the direction of the solution provided by the AI tools. To improve the experience, different organisations are working on improving prompt creation and debugging.
Conclusion
When it comes to web3 security and auditing, AI tools are a help, there is no doubt about that, but the question is, is that enough? the answer is a big “NO”. As discussed, some of the crucial vulnerabilities can easily get left out, and there is a huge possibility of false alarms. These false alarms create a false sense that chatGPT can identify all the bugs and lead the user to believe in it, but the reality is different and can be harsh if we get dependent on AI tools only.
AI may become very effective, but we have a long way to go. The best way we can improve security is by using both AI and manual coverage of the security aspects of smart contracts.
Regarding smart contract security, there is no replacement for audits. It is of utmost necessity to go for an audit, and without auditing, there can never be trust among users, as audit reports mean a lot. Many users look for the audit report before trusting the projects. One of the leading firms in auditing services is QuillAudits. With 700+ projects secured and many more coming, we ensure the complete safety of the protocols. Check out our website now and get your project audited.
