Teraswitch failure leaves 29% of verifiers offline, and Solana network survives disruption

Solana nearly suffered network failure, 29% of authentication nodes went offline due to a Teraswitch failure.

This week, Solana's network was nearly shut down due to a massive network outage. The reason was that a routing failure in the Teraswitch data center caused nearly 29% of pledged tokens to go offline. This incident has exacerbated industry concerns about blockchain's over-reliance on centralized infrastructure providers.

The failure started in Miami and spread around the world

The technical failure started in Teraswitch's data center in Miami, where engineers discovered that the center had declared a wrong route. The misconfiguration spread through route reflectors in Amsterdam and ultimately affected multiple Teraswitch facilities in Europe and Asia. The major cities affected include London, Amsterdam, Dublin, Frankfurt, Singapore and Tokyo, all with complete network path outages. The North American data center maintained normal operations.

Pledge service provider Marinade reported that approximately 90 verification nodes lost contact during the incident, and these nodes pledged a total of 28.83% of SOL tokens. Although Solana's offline pledge volume was once close to 20 million SOL, just a step away from the threshold of 33.34% that triggered network paralysis, transaction processing and block production were not interrupted.

In decentralized blockchain, final certainty refers to the stage at which a transaction is irreversible. If more than one-third of the pledged tokens are offline at the same time, the blockchain will lose its ability to finalize transactions, causing all user operations to be suspended.

Small Dictionary: Marinade is a Solana-based liquidity pledge protocol that allows users to pledge SOL and exchange it for a tokenized version, making online pledge more accessible and flexible.

Teraswitch engineers resolved the root cause in approximately 10 minutes, and all connections were restored at 4:16 UTC. In total, the affected verification nodes lost 333SOL pledge rewards, but Marinade said it will compensate operators through its verification node bond process.

Centralization and verification node centralization issues raise concerns

Marinade pointed out that the fundamental risk factor in this incident lies in the centralization of pledges. According to its analysis, a single network operator (ASN number AS20326) held 27.34% of the total pledged SOL during the peak failure period, exceeding the 25% recommended security cap within Solana. During the incident, 94% of SOLs related to the carrier were offline.

In addition, approximately 14.1 million SOLs disappeared from verification nodes hosted by other providers such as Latitude.sh, Limestone, Butterfly Research, and Allnodes. Marinade said it was unclear whether these secondary issues were directly related to the same routing failure or were separate incidents.

The pledge platform also admitted that it currently faces serious infrastructure centralization challenges. Marinade revealed that only four autonomous system numbers bear two-thirds of its entrusted pledge. In hindsight, the Marinade team bluntly said: "No one should feel at ease about this, including ourselves."

The Solana Foundation responded by defending its architecture.

Jacob Creech, vice president of technology at the Solana Foundation, said that this failure just proves the resilience of the core design of blockchain. Of the 699 pledge verification nodes, 597 were able to continue voting operations throughout the event. All offline verification nodes return to service within 40 minutes of the failure.

Creech emphasized on the X platform that because Solana verification nodes are distributed among independent infrastructure providers, a single provider failure did not disrupt network operations.

Currently, Solana has locked in a total value of US$4.3 billion in decentralized financial agreements. Despite this, the network has had multiple failures in its history. For example, a major accident in February 2024 took nearly five hours for the entire system to resume normal operation.

As of publication of this article, Solana was trading at approximately US$75.79.