Beyond the Hype: ChatGPT and Smart Contract Auditing

Beyond the Hype: ChatGPT and Smart Contract Auditing

Beyond the Hype: ChatGPT and Smart Contract Auditing

Beyond the Hype: ChatGPT and Smart Contract Auditing

Beyond the Hype: ChatGPT and Smart Contract Auditing

Read Time: 6 minutes

Exploring the effectiveness of chatGPT in Smart Contract Auditing

It was 30th November, 2022, when ChatGPT was launched. It did not take it long to take the world by storm. No matter what social media you use, there are posts, memes, informative articles and whatnot on chatGPT. Not only that, the chatGPT was the talk of the mainstream media. There are no second thoughts when I say everyone talked about chatGPT and its power.

In this blog, let’s discuss how chatGPT is used or can be used in smart contract auditing or Web3 cyber security. Let’s first start with what chatGPT exactly is.

What is ChatGPT?

ChatGPT is an interactive chatbot that takes in prompts and returns answers based on its trained data. It has a remarkable ability to communicate in conversational dialogue and provide responses that can seem surprisingly human.

Apart from that, one of the things that make it smarter is its unique ability to keep on learning from the user’s input data; this is implemented in a layer of Reinforcement learning with human feedback (RLHF), which helps it return answers that are satisfying to humans. 

Training data

Every AI model is nothing but a trained machine that gives answers based on its learning and findings from the training data. The training data can be anything from videos to text which is fed to a model which learns about this data, and when a problem is proposed to this model, based on its learning from the training data, it gives answers. 

The chatGPT was trained on the data collected from the internet, including sources such as Reddit discussions, to help ChatGPT learn dialogue and achieve a human-like response style. chatGPT is also trained on human feedback. This technique is called Reinforcement Learning with Human Feedback so that the AI ​​learns what people expect when they ask a question.

ChatGPT can find Vulnerabilities

Long after its release, people started experimenting with the capabilities of chatGPT in various use cases and scenarios. This experimentation was also done in smart contract security.

And chatGPT sure did not fail us. However, it still has room for improvement, but it proved to be useful and of significant help to auditors and people dealing with smart contracts. When it comes to the well-known hacks and some hacks which have been in the system for quite some time now, it is very useful in catching them.

Some of the common vulnerabilities that chatGPT finds with a bit of accuracy are:-

  1. Renterancy attack: This is a common vulnerability in which an attacker can repeatedly call a function within a smart contract before the previous execution has been completed, leading to unexpected or malicious behaviour.
  1. Integer Overflows/Underflows: Smart contracts often rely on integer calculations, and if these calculations are not properly checked, they can result in unexpected or incorrect behaviour.
  1. Unchecked return values: A contract may not properly handle unexpected return values from external calls, which may lead to a potential vulnerability and can cause harm.
  1. Unprotected functions: A contract may not have proper access control, leading to unauthorised access to sensitive functions. Which can lead to heavy loss.

There are some other vulnerabilities and issues that chatGPT can identify with smart contracts, and you will surely be surprised to see them. Still, through our tests, we discovered that you would often receive a false alarm, and there is a huge possibility that some crucial bugs are missed.

Can chatGPT find all vulnerabilities?

While chatGPT is a useful tool and a breakthrough of AI for the masses, it is still far from perfect and cannot be left to completely secure smart contracts.

Our test found that the chatGPT raised a false alarm for a re-entrance attack, which was already guarded and tested. Apart from that, there were some more false alarms, and most importantly, the critical bug our team found was completely ignored by chatGPT. Let’s discuss some of the things chatGPT is likely to miss.

  1. Project-specific Logic:- The project’s backbone is its logic and how things are interconnected, but chatGPT seems to miss on it. During tests, it was found that chatGPT was often unable to find the critical bug, which was logic specific. Due to the complexity of the underlying infrastructure of the protocol, chatGPT misses the critical vulnerabilities that arise due to the interconnection of contracts to fulfil the logical requirement of the project.
  1. Inaccurate math calculation and statistical models:- When it comes to projects, whether it’s a gaming project, a DeFi project or can be anything, it mostly involves mathematical calculations and relations. These formulae are often left unchecked and unmonitored by the chatGPT, and potential bugs are missed.
  1. Irregularities in Intended design and Implementation:- Many times, the implementation by the developers is not as correct as it should be, leading to security issues. This has been exploited in the past and continues to be one of the essential sectors that can be improved, and chatGPT is a bit ignorant on this front as well.

Why can’t chatGPT do it all?

No doubt, it can be of great aid to the developers. Still, it can only partially be trusted with smart contract auditing as it clearly needs to cover up all the vulnerabilities and leave some of the crucial ones unchecked, which can be devastating. In this section, let’s discuss why chatGPT is not ready yet.

The chatGPT needs to reason well about some of the higher-level topics. It may produce inconsistent results or provide incomplete solutions in such cases. Ownership of contracts, re-entrance, and fee distribution are some of the higher-level topics that chatGPT needs to grasp most of the time.

Integrating the Large Language Models(LLM) with the software is something which needs to be worked upon. New advancements are trying to improve in this field, but we are still looking for a breakthrough. This field is relatively new, so we will see a good investment to improve LLM’s performance. Different tools need to create, which will help developers interact better.

Prompt creation is another thing that developers need to find their way through sometimes. Prompts are really crucial when working with the LLMs. The prompt dictates the direction of the solution provided by the AI tools. To improve the experience, different organisations are working on improving prompt creation and debugging.

Conclusion

When it comes to web3 security and auditing, AI tools are a help, there is no doubt about that, but the question is, is that enough? the answer is a big “NO”. As discussed, some of the crucial vulnerabilities can easily get left out, and there is a huge possibility of false alarms. These false alarms create a false sense that chatGPT can identify all the bugs and lead the user to believe in it, but the reality is different and can be harsh if we get dependent on AI tools only.

AI may become very effective, but we have a long way to go. The best way we can improve security is by using both AI and manual coverage of the security aspects of smart contracts.

Regarding smart contract security, there is no replacement for audits. It is of utmost necessity to go for an audit, and without auditing, there can never be trust among users, as audit reports mean a lot. Many users look for the audit report before trusting the projects. One of the leading firms in auditing services is QuillAudits. With 700+ projects secured and many more coming, we ensure the complete safety of the protocols. Check out our website now and get your project audited.

2,763 Views

Blockchain for dog nose wrinkles' Ponzi makes off ~$127M🐶

Project promised up to 150% returns on investment in 100 days, raising about 166.4 billion South Korean won — or about $127 million — from 22,000 people.

Latest blogs for this week

Understanding Fuzzing and Fuzz Testing: A Vital Tool in Web3 Security

Read Time: 5 minutes When it comes to smart contracts, ensuring the robustness and security of code is paramount. Many techniques are employed to safeguard these contracts against vulnerabilities
Read More

How EigenLayer’s Restaking Enhances Security and Rewards in DeFi

Read Time: 7 minutes Decentralized finance (DeFi) relies on Ethereum staking to secure the blockchain and maintain consensus. Restaking allows liquid staking tokens to be staked with validators in
Read More

ERC 404 Standard: Everything You Need to Know

Read Time: 7 minutes Introduction Ethereum has significantly shaped the crypto world with its introduction of smart contracts and decentralized applications (DApps). This has led to innovative developments in
Read More

DNS Attacks:  Cascading Effects and Mitigation Strategies

Read Time: 8 minutes Introduction DNS security is vital for a safe online space. DNS translates domain names to IP addresses, crucial for internet functionality. DNS ensures unique name-value
Read More

EIP-4844 Explained: The Key to Ethereum’s Scalability with Protodanksharding

Read Time: 7 minutes Introduction  Ethereum, the driving force behind dApps, has struggled with scalability. High fees and slow processing have limited its potential. They have kept it from
Read More

QuillAudits Powers Supermoon at ETH Denver!

Read Time: 4 minutes Calling all the brightest minds and leaders in the crypto world! Are you ready to build, connect, and innovate at the hottest event during ETH
Read More

Decoding the Role of Artificial Intelligence in Metaverse and Web3

Read Time: 7 minutes Introduction  Experts predict a transformative shift in global software, driven by AI and ML, marking the dawn of a new era. PwC predicts AI will
Read More

Transforming Assets: Unlocking Real-World Asset Tokenization

Read Time: 7 minutes In the blockchain, real-world assets (RWAs) are digital tokens that stand for tangible and conventional financial assets, including money, raw materials, stocks, and bonds. As
Read More
Scroll to Top

Become a Quiffiliate!
Join our mission to safeguard web3

Sounds Interesting, Right? All you have to do is:

1

Refer QuillAudits to Web3 projects for audits.

2

Earn rewards as we conclude the audits.

3

Thereby help us Secure web3 ecosystem.

Total Rewards Shared Out: $200K+