Why did the Solana blockchain halt again?

• Solana blockchain halted multiple times.

• This time, Solana's consensus mechanism malfunctioned due to a bug.

• The bug was known to the team, but it didn't cause problems in the past.

The value of Solana, a cryptocurrency in the top 10 in terms of market capitalization, dropped by more than 12% on June 1 as a result of its second outage in a month. A problem with the blockchain's processing of a niche transaction type meant for offline use caused it to malfunction.

After its validators stopped processing new blocks, all SOL-based applications were shut down for over four hours. 

Investors have increasingly turned to Solana and other alternative blockchains for diversification of their portfolios since last year's crypto bull run, but as the blockchain has been hit by multiple outages, their confidence in it has diminished.

The protocol is branded as a beta version of the mainnet, so bugs and errors are still possible. This year, Mainnet-Beta blocks were halted twice: on 30 April and 1 June. Previously, the incident occurred in September 2021 and December 2020 – four times in Solana's 26-month existence. 

What exactly caused the outage?

A spokesperson for Solana Labs, Austin Federa, confirmed that validators restarted only after durable nonce transactions, a method by which transactions can be signed offline without requiring a recent block hash, were disabled. These niche transactions were double counted by the network's validators at two different block heights instead of being treated as one transaction. This resulted in a breakdown of Solana's consensus mechanism.

“This was probably a bug that existed for a while but never really became an issue because it isn't something that most people use,” Federa explained. 

Crypto exchanges have been using these types of transactions more extensively recently, possibly because of their cold storage setup. 

The problem persisted even after the restart, with exchanges reporting failures with Solana deposits and withdrawals.

Laine from Stakewiz, the Solana validator operator, said that the bug had been known and was being fixed before the event.

In a Twitter Spaces conversation, Austin Federa and Solana Labs co-founder Anatoly Yakovenko discussed whether this bug that took down the network was linked to the other performance improvements and features that the team had been working on. Anatoly confirmed that this was one of those totally unrelated problems.

Are Solana's overzealous plans to blame for the outage?

While Solana is having these issues, its founders take time to criticize other blockchains instead of addressing their own.

Two co-founders of Solana, Anatoly Yakovenko and Raj Gokal, claimed during a podcast that Cardano is being built "too carefully". "This is why they are never gonna ship," Anatoly Yakovenko said, instead of "shipping the code and getting paid for it."

As Cardano Co-Founder Charles Hoskinson replied to this criticism, he admitted that ADA developers are indeed diligent about building their software, since "millions of users and thousands of businesses" rely on the infrastructure they are building.

Hoskinson concluded that Solana's team of software developers should develop something as worthwhile as a hospital based on its accelerated approach to software development.

A down-to-earth analysis of Solana's performance

Galaxy Digital's Surveying Solana report states that Solana's whitepaper claims a theoretical throughput of 710,000 transactions per second, yet according to Solana's website, it is averaging ~1,500 transactions per second (tps). So, Solana's ideal future is 500 times different from the current reality.

It would be incorrect to take Solana's self-reported tps at face value because it counts internal consensus messages as transactions, which is not standard practice for the blockchain industry.

Solana calls consensus messages "vote transactions" - these are validators managed by voting accounts that handle vote registration, vote collection, and new vote signing. Solana transactions involving smart contract interactions with dapps are called "non-vote" transactions (most other blockchains only factor in "non-vote" transactions).

Source: Galaxy Digital Research 

According to Dune Analytics data, vote transactions account for 80-90% of all of Solana's transactions. By subtracting this "consensus overhead" from Solana's reported 1,500 transactions per second, we arrive at a true tps of about 300 non-voting tps rate (although this number is constantly in flux).

Although it is a far cry from its theoretical 710,000 tps cited in its whitepaper, it is still an impressive number compared to what Ethereum can achieve with its 12-15 tps limit. 

Centralization issue 

The proof-of-stake network of Solana is secured by 1,785 validators as of June 6th, 2022.

The stake distribution of these validators is relatively concentrated; therefore, they are theoretically able to halt the network completely on their own. It is also worth mentioning that this issue of centralization is not unique to Solana (the Avalanche network contains similar levels of stake across its validators).

Custodians and centralized exchanges have large balances of Solana tokens through their daily operations. By running their own validators, they can earn additional yield from these tokens. Centralized exchanges can instantly vote on and propose blocks to the Solana network by pointing their massive Solana balances to their own validators. In other words, if a centralized exchange has a lot of Solana on its balance sheet, it can run a very profitable validator operation with little effort. 

A user can also delegate their stake in Solana tokens to a popular validator, this is trivially easy from the point of view of a retail user. Stake delegation is even part of some centralized exchanges' product offerings, and they offer users additional yield on their static token balances in exchange for cutting a piece of the stake rewards. As a result, the "rich get richer," as the most powerful validators will be able to draw more capital and grow their influence over the network.

Conclusion 

Solana is a fast-growing blockchain that is becoming more popular in many fields and on track to beat Ethereum as a smart contract leader. Despite fundamental design flaws, technological fixes and upgrades could mitigate or resolve these problems. Fast doesn't always mean better, so maybe the Solana team should learn from other blockchain projects that are too careful with their development because they realize how vital they are to the crypto community and industry in general.