What is Antithesis? How we’re different How it works See an example
Problems we solve Case studies Working with Antithesis
Backstory Leadership Careers Brand
Pricing Docs Blog
Let's talk

Testing the Ethereum merge

In September 2022, the Ethereum network completed “the Merge,” an ambitious project that shifted Ethereum from “Proof of Work" to “Proof of Stake.” The Merge provided multiple benefits—chief among them an over 99% reduction in energy consumption across the network—but it also came at a significant risk. New bugs or security vulnerabilities could have been introduced that would have damaged the stability of the Ethereum network and the trust of its users.

A year before the Merge, the Ethereum Foundation began working with Antithesis to test the Merge codebase under very demanding conditions, in order to ensure that even rarely-triggered bugs would be found and fixed before the Merge was initiated.

About Antithesis

Antithesis is a continuous reliability platform that autonomously searches for problems in whole software systems. This search occurs within a simulated environment where every problem found can be perfectly reproduced, allowing for efficient debugging of even the most complex problems. Antithesis was founded by the creators of FoundationDB, an exceptionally reliable distributed database that is the underpinning of cloud infrastructure at Apple, Snowflake, and other companies. Antithesis is inspired by the powerful simulation testing framework that made FoundationDB possible.

About Ethereum

Ethereum is a decentralized blockchain platform featuring smart contract functionality, proposed by Vitalik Buterin in 2013, and launched in 2015. Its core feature is the Ethereum Virtual Machine (EVM), which enables the execution of smart contracts for automating transactions, managing digital assets, and running decentralized applications (dApps). Ethereum's native cryptocurrency is Ether (ETH), which is the second-largest cryptocurrency by market capitalization. The Ethereum network serves as the foundation for many other financial products and crypto projects across the globe.

About the merge

The Ethereum Merge was a significant upgrade to the Ethereum blockchain, aimed at addressing a key problem: the environmental impact and inefficiency of the Proof of Work (PoW) consensus mechanism.

Ethereum's original PoW mechanism, similar to that of Bitcoin, required extensive computational power due to its reliance on cryptocurrency mining—which used energy intensive computing to create new cryptocurrency tokens and power the network. This high energy use was of major concern due to its impact on the carbon footprint of the Ethereum network. The PoW system was also less scalable, and was limited in processing speed and transaction capacity.

The Merge transitioned Ethereum from PoW to Proof of Stake (PoS) for consensus. In PoS, validators are chosen to create new blocks and confirm transactions based on the amount of cryptocurrency they are willing to stake as collateral, rather than on computational power. This change was part of Ethereum's long-term scaling strategy.

If successful, the Merge would reduce Ethereum’s energy consumption by more than 99%. This reduction in energy use would address environmental concerns surrounding the energy usage of blockchain technology. The day before the Merge, Vitalik Buterin (creator of Ethereum) tweeted:


This would make the Merge one of the biggest decarbonization events in history. Additionally, PoS could offer improved security, better scalability, and a foundation for further upgrades that could enhance transaction speed and efficiency. In the months leading up to the Merge, Vitalik told an Ethereum conference that the Merge’s completion would mean that the progress on Ethereum’s roadmap was more than half complete, and that:

At the end of this road map, Ethereum will be a much more scalable system. By the end, Ethereum will be able to process 100,000 transactions per second.

However, the process of implementing the Merge involved substantial risk, especially as it would not be practical to “roll back” the Merge if there were problems encountered after it was implemented. The technical challenge of seamlessly transitioning the multi-billion dollar Ethereum network without disrupting its ongoing operations was very difficult, and very important to get right. Additionally, any bugs or security vulnerabilities in the new PoS system could compromise network stability, speed, or security—possibly causing loss of funds and lack of trust in the network.

The day before the merge, in an article titled “Crypto Investors Step Up Bets Against Ether as ‘Merge’ Looms” The Wall Street Journal reported:

It’s very difficult for me to see how a fully functioning platform with a market cap of around $200 billion can essentially change the engines out in flight and not have some sort of security issue in all that complexity,” said Christopher Calicott, a crypto venture investor at Trammell Ventures.

Bugs in the code and other network vulnerabilities could remain undiscovered until the rollout. The Ethereum Foundation, a nonprofit supporting the blockchain, offered up to $1 million for information on any critical bugs discovered on the network before Sept. 8.

Needless to say, while the potential benefits both to the Ethereum network and to the environment were great, so were the potential risks if problems were introduced in the process of the Merge.

How Antithesis worked with Ethereum

The Ethereum Foundation has always prioritized extensive testing; their internal test system “Hive” runs 44,000 unit tests a day, and Ethereum also employs some of the world’s leading security engineers. However, the complexity of properly implementing the Merge (once described as “trying to change the tires on a car while driving down the highway”) was such that the standard approach of manually written tests was deemed to be insufficient.

The Ethereum Foundation began working with Antithesis about a year before the Merge occurred, drawn by Antithesis's unique ability to explore unexpected execution paths of complex software systems inside of a simulated environment, with perfect reproducibility. After a successful proof of concept, testing work began in earnest—marking the first time a cryptocurrency network had ever been tested in this way.

Antithesis worked with the Ethereum Foundation team to package their system for the Antithesis environment, create a realistic workload, and define the expected properties of their system. Antithesis ran Ethereum’s entire network (including the proposed Proof of Stake code) inside of the Antithesis environment. Antithesis stress-tested this simulation of the Ethereum network by injecting a variety of faults (network issues, hardware failures, frozen processes, etc.), which were able to drive the Ethereum network into states that had never before been achieved by Ethereum’s testing tool, but which might occur in the real world.

As the Merge codebase evolved over the next year, and as more of the eight total Ethereum execution and consensus clients were added, Antithesis continuously tested new versions of the code and reported any newly discovered issues. As new issues were found, Antithesis worked with the Ethereum Foundation team to identify which of them represented critical bugs.

Antithesis then provided the EF team with perfect reproductions of these test runs—including debugging artifacts such as log dumps, core dumps, and even interactive debugging sessions. This enabled EF to get to the root cause of newly-identified issues very quickly, and to propose fixes to them. Once a code change had been proposed to fix a given bug, the fix was then tested again with Antithesis, using similar conditions to those that had triggered the original bug. In this way EF was able to validate both that the fix had been successful, and that it had not introduced new bugs.

Summary of bugs found & resolved

Antithesis’s continuous testing process led to the discovery and resolution of dozens of serious bugs within the Ethereum network’s codebase prior to the Merge. We summarize a few representative ones below:

Panics, crashes or denials of service

Antithesis identified thirteen bugs within the category of “Panics, Crashes, or Denials of Service”. These types of bugs are important to fix because they could cause entire groups of Ethereum nodes to go down, affecting the resilience of the network. One example of this kind of bug was:

  • PR3813 - An issue where a system failed to create attestations for upcoming blocks in its early attester cache. This issue had the potential to cause a missed head vote if the system was asked to generate an attestation for a slot earlier than the current head block, typically occurring shortly after verifying a block and setting it as the head.
Blockchain operations

Antithesis identified twenty-two bugs that affected the overall performance of the Ethereum blockchain, including Fork Choice functionality, plus several other, non-critical errors. These types of bugs are important to fix because they affect the operation of Ethereum nodes. Bugs were found across all six programming languages used to implement the eight clients being tested. Some examples of bugs in chain operations include:

  • PR10994 - An oversight in error handling for invalid blocks, causing them to become trapped in the pending queue. This led to an Ethereum node executing resource-intensive and unnecessary block validations, hampering system efficiency and causing computational overhead.

  • PR 11024 - A missing nil check, impacting communication between the execution and consensus client. This led to an update attempt, even though the payload ID was unexpectedly nil.

  • PR 10991 - A missing check for the invalid_block_hash error from the Execution Environment (EE). The client would fail to remove the block and its associated state from the database.

These are some examples of bugs that were only found due to Antithesis. Each of them was fixed with debugging help from Antithesis before they could negatively affect the Merge.

Success!

As the tech and crypto world waited in anticipation, the Merge was successfully executed on the evening of September 15th. The Antithesis team watched it happen together with Ethereum leadership at a private Merge party in Denver, and all breathed a sigh of relief as no issues were encountered, and as the network continued operating without interruption. The Proof of Stake system was even more energy efficient than expected, resulting in a 99.95% decrease in energy consumption compared to the previous Proof of Work implementation. This bold and successful effort not only improved the Ethereum network, it set the stage for a more environmentally friendly crypto industry.

Antithesis is an exciting and unique tool for debugging blockchains and distributed systems. We used it extensively when testing the Merge. Antithesis was able to deterministically explore and find bugs in very exotic states and scenarios, ones that would have been nearly impossible to hand-code and unlikely to be hit in less stateful, traditional fuzzing.

Danny Ryan, Researcher

Antithesis allowed us a unique way to test and debug the Ethereum Merge. The ability to deterministically reproduce errors helped greatly improve the time to find the root cause of an issue and their approach of fuzzing allowed us to push Ethereum clients to edges that would otherwise be extremely hard.

Paritosh Jayanthi, DevOps Engineer

Ongoing testing

Antithesis’s work with the Ethereum Foundation continues to this day—new iterations of the Ethereum codebase continue to be tested and improved with Antithesis’s help. One notable moment occurred in early 2023 when the Ethereum network briefly stopped finalizing transactions. After this occurred, Antithesis was able to quickly replicate the “finality death spiral,” confirming Ethereum’s hypothesis of the conditions that could lead to this condition again.

Conclusion

The successful completion of the Ethereum Merge in September 2022 marked a watershed moment in the history of blockchain and cryptocurrency. The collaboration between the Ethereum Foundation and Antithesis played a pivotal role in ensuring the smooth transition of the Ethereum network from Proof of Work to Proof of Stake. Through Antithesis's unique testing capabilities, numerous potential vulnerabilities and bugs were identified and rectified before they could impact users, securing the network's stability and reliability. The Merge has set a new standard for blockchain technology, proving that such ambitious upgrades can be achieved successfully, and paving the way for a more sustainable and efficient future in the crypto industry. As Ethereum continues to evolve, its ongoing partnership with Antithesis ensures that the network remains reliable and resilient, ready to face future conditions—whether anticipated or not.