How a Critical Bug in Solana Network was Detected and Timely Patched

In April, our vulnerability detection system discovered an issue in the rBPF of Solana (i.e., the virtual machine where all the Solana dApps are running on: https://github.com/solana-labs/rbpf). After a careful investigation, we found that it is a security vulnerability that can lead to a wrong execution path of a contract. We have reported the bug to the Solana security team, and the team immediately confirmed and fixed the vulnerability. Also, they granted our team a $800,000 USD worth of SoL token.

The vulnerability exists in newer versions of rBPF (0.2.26 to 0.2.27). When we reported the issue, the validators used on the mainnet has not been upgraded to the affected version(s). Our system detected the issue prior to the affected version being merged, making the validator of the mainnet immune to this vulnerability.

We elaborate on the details of this vulnerability in the following.

1. eBPF and rBPF

eBPF (Extended Berkeley Packet Filter) was initially developed for filtering packets in the kernel. Due to the security, efficiency, and scalability of eBPF, it is now used in various areas like networking, tracing, profiling, etc. Considering the rich capability of eBPF, Solana uses it as the execution engine for the smart contract. To build the dApps on Solana, developers develop their smart contracts in Rust and compile the contract into eBPF bytecode.

Solana uses the rBPF, a virtual machine written in Rust, to execute the compiled BPF bytecode. However, whether the proposed virtual machine (i.e., rBPF) is robust, secure, and precise is unknown. If security issues exist inside the rBPF, all the validators containing the rBPF can be influenced, resulting in the huge loss (e.g., Loss of Funds) for the whole Solana network.

2. Root Cause

We developed a tool that can automatically locate the implementation bugs of rBPF and scan the code of rBPF periodically. One serious problem was identified in rBPF (version 0.2.26) during the scanning process, which may lead to the wrong execution path of a contract.

Specifically, the sdiv instruction is used as a signed division instruction, introduced as a feature enabled by default in rbpf 0.2.26. sdiv supports division for operands in both 32-bits (i.e., sdiv32) and 64-bits (sdiv64). For instruction sdiv32, the calculation result is stored in the bpf register, which is a 64-bits one. However, if the instructions after the sdiv32 one read the calculation result as 64 bits, the result may differ. It is because rBPF does not extend the calculation result of sdiv32 into the correct value in 64-bits during the JIT compilation.

For example, if a positive number (i.e., 12) is divided by a negative number (i.e., -4) with sdiv32, the correct result should be -3 in both 32 bits and 64 bits. The code below is an example.

After running and tracing it under both JIT and Interpreted modes, we could observe the differences between them:

2.1 Interpreted Mode

0 [0000000000000000, 0000000400000000, 0000000000000000, 0000000000000000, 0000000000000000, 0000000000000000, 0000000000000000, 0000000000000000, 0000000000000000, 0000000000000000, 0000000200014000]    29: lddw r5, 0x10000000c
 1 [0000000000000000, 0000000400000000, 0000000000000000, 0000000000000000, 0000000000000000, 000000010000000C, 0000000000000000, 0000000000000000, 0000000000000000, 0000000000000000, 0000000200014000]    31: sdiv32 r5, -4
 2 [0000000000000000, 0000000400000000, 0000000000000000, 0000000000000000, 0000000000000000, FFFFFFFFFFFFFFFD, 0000000000000000, 0000000000000000, 0000000000000000, 0000000000000000, 0000000200014000]    32: jslt r5, 0, lbb_7
 3 [0000000000000000, 0000000400000000, 0000000000000000, 0000000000000000, 0000000000000000, FFFFFFFFFFFFFFFD, 0000000000000000, 0000000000000000, 0000000000000000, 0000000000000000, 0000000200014000]    36: exit

2.2 JIT Mode

0 [0000000000000000, 0000000400000000, 0000000000000000, 0000000000000000, 0000000000000000, 0000000000000000, 0000000000000000, 0000000000000000, 0000000000000000, 0000000000000000, 0000000200014000]    29: lddw r5, 0x10000000c
 1 [0000000000000000, 0000000400000000, 0000000000000000, 0000000000000000, 0000000000000000, 000000010000000C, 0000000000000000, 0000000000000000, 0000000000000000, 0000000000000000, 0000000200014000]    31: sdiv32 r5, -4
 2 [0000000000000000, 0000000400000000, 0000000000000000, 0000000000000000, 0000000000000000, 00000000FFFFFFFD, 0000000000000000, 0000000000000000, 0000000000000000, 0000000000000000, 0000000200014000]    32: jslt r5, 0, lbb_7
 3 [0000000000000000, 0000000400000000, 0000000000000000, 0000000000000000, 0000000000000000, 00000000FFFFFFFD, 0000000000000000, 0000000000000000, 0000000000000000, 0000000000000000, 0000000200014000]    33: lddw r0, 0x1
 4 [0000000000000001, 0000000400000000, 0000000000000000, 0000000000000000, 0000000000000000, 00000000FFFFFFFD, 0000000000000000, 0000000000000000, 0000000000000000, 0000000000000000, 0000000200014000]    35: exit

In the Interpreted mode, register r5 is set to 0xFFFFFFFFFFFFFFFD (-3 in both 32-bit and 64-bit mode), while in JIT mode, r5 is set to 0x00000000FFFFFFFD. In this case, r5 will be recognized as a positive number (i.e., 0x00000000FFFFFFFD) for instruction jslt, which receives a 64-bit value. After that, the execution path will be totally wrong.

3. Impact

This wrong implementation can lead to the incorrect execution path of a contract and may cause serious problems.

For instance, if an essential operation in a smart contract relies on the result of sdiv32 instruction, it can lead to incorrect execution results and be abused by attackers.

This issue was introduced in https://github.com/solana-labs/rbpf/pull/283, meaning that rBPF is vulnerable from version 0.2.26. We identified the problem and reported it to the Solana security team on 2022 April 28th. The team quickly responded to our report and fixed the issue in a few hours by adding the sign-extend operation for sdiv32 instruction. The fix is in https://github.com/solana-labs/rbpf/pull/310. Due to the timely detection and report from our team, validators of the mainnet are not affected by this vulnerability.

This issue is classified as a protocol liveness bug, leading to $800,000 bug bounty granted by Solana.

Timeline

• 2022/04/28: We reported the problem to the Solana security team
• 2022/04/29: the vulnerability was fixed
• 2022/05/09: CVE-2022-23066 was assigned
• 2022/06/01: The bug bounty was granted

About BlockSec

The BlockSec Team focuses on the security of the blockchain ecosystem, and collaborates with leading DeFi projects to secure their products. The team is founded by top-notch security researchers and experienced experts from both academia and industry. They have published multiple blockchain security papers in prestigious conferences, reported several zero-day attacks of DeFi applications, and released detailed analysis reports of high-impact security incidents.

Twitter: [BlockSecTeam]

Medium: https://blocksecteam.medium.com

Website: https://www.blocksec.com