Solana outage caused by a previously identified bug, devs say

A post-mortem report on this week’s Solana network outage pointed the finger at a previously known bug.

The network was unusable for five hours Tuesday. Validators — a key infrastructure layer of Solana, responsible for running the software that adds and verifies transactions to the blockchain — later restarted the network before 10 am ET.

The post-mortem, penned by Anza, concluded that an infinite loop as a result of a bug caused the validators to stall on a certain block. An infinite loop leads to validators endlessly repeating an action, and with validators put out of action, that prevents the network from processing more transactions.

Read more: Solana price dips as outage requires a network restart

Specifically, Solana’s Just-in-Time (JIT) compilation cache encountered a bug, leading to an infinite loop of recompilations for some older programs, thus monopolizing network resources and halting operations.

As a result, the compiler — meant to enhance performance by compiling code in real time — became a bottleneck. To resolve this, the Solana team deployed a fix in a new release, addressing the bug and preventing such loops.

The issue was “consistent with a bug that had been identified during the investigation of a recent Devnet outage and for which a patch was to be imminently deployed.” ​​The problem was first flagged to the Solana security team in April 2022.

Read more: Solana’s DeFi users mostly unbothered by network outage

“This fix eliminates the ability to create the preconditions required to trigger the bug, which was a simpler resolution. A more complete fix will be included with further improvements to LoadedPrograms and allowed to stabilize with the regular release cycle,” the report explained.

The outage was the sixth major outage incurred by Solana since its inception. Solana’s last outage took place in February of last year.

As Blockworks’ David Canellis noted at the time, the price of SOL — Solana’s native asset — dipped briefly on the news of the outage. But it recovered swiftly, and according to market data, the event indicated that market responses to Solana network outages have become more muted over time.

Source