Tiny Rounding Down, Big Fund Losses: An in-depth analysis of the recent Balancer incident

Updated on September 15, 2023: Balancer has published the official post-mortem, which provides a detailed description about the whole story of this incident, including the experience and lessons learned. This post-mortem, with its intricate and excellent narrative, is compelling and certainly worth your time.

From a security perspective, this post-mortem reveals that there exist two bugs. The first one is the rounding down error we have discussed in our report, and the second one is the “resets rate on 0 supply”, which occurred in attack steps 3.6 and 3.7, as described in our report. Balancer’s report regards the second one as the most critical issue, with the first being contributory. However, we believe both bugs are equally important for profitable exploitation:

The first bug is used to pump the token rate, serving as the root cause of the profit. Without it, generating profit would be unfeasible.

The second bug enables the exploit by balancing the bb-a-tokens’ debt. Without it, the attack would fail due to the poor liquidity of the bb-a-tokens, given there are no other sources to obtain these tokens (unless the attacker manages to get them by some means).

On August 22, 2023, Balancer publicly announced the presence of a critical vulnerability affecting multiple boosted pools, and urged users to withdraw LPs from the affected pools immediately. Balancer had initiated emergency mitigation procedures to secure the majority of TVL, yet some funds remained at risk. Unfortunately, on August 27, five days later, we noticed several attacks in the wild. Since then, assets exceeding $2.12M have been stolen.

As of the time of writing this report (more than three weeks after the announcement, at a point we believe it is safe to do so), Balancer has not released any in-depth analysis of this vulnerability. In this report, we aim to provide a comprehensive analysis, mainly based on one of the attack transactions.

Key takeaways (TL;DR)

Our investigation indicates that the root cause stems from the price manipulation resulting from the rounding down logic in the linear pool. This consequently affects the cached token rate used by the corresponding boosted pool inappropriately.
This incident emphasizes the critical need for prompt notifications to projects that have forked from a vulnerable source, which indeed poses a significant challenge for the whole community.
The numerous ongoing attacks underscore the necessity of proactive threat prevention, which could inevitably aid in mitigating prospective losses.

In the following sections, we will first provide some essential background information about Balancer. Following that, we will conduct a comprehensive analysis of the vulnerability and the associated attack. Finally, we will provide a brief summary of attacks we have observed thus far, along with their corresponding profits.

0x1 Background on Balancer

Balancer V2 [1] is a decentralized automated market maker (AMM) protocol representing a flexible building block for programmable liquidity. Unlike other AMMs where token accounting is paired with pool logic, Balancer separates the token accounting and management from the pool logic, which can improve swap efficiency by reducing lots of token transfers.

Balancer supports various types of pools. Each pool is associated with an LP token named BPT (i.e., Balancer Pool Token). Basically, BPT value is calculated based on the total value of all underlying tokens.

Balancer supports multi-hop swaps, also known as batch swaps, that leverage the best prices from all pools registered with the Vault. Specifically, the Vault provides the batchSwap function to facilitate multi-hop swaps.

A flash swap in Balancer's pools eliminates the need to hold any of the input tokens traditionally required to execute a swap. Instead, upon identifying an imbalance, you can instruct the Vault to execute the swap and subsequently receive the reward.

0x1.1 Various Pools in Balancer

In the following, we briefly introduce some concepts of pools which are relevant to this vulnerability.

Linear Pools: Linear pools [2] are Balancer pools that facilitate the exchange of an asset and its wrapped, yield-bearing counterpart at a known exchange rate. As the name suggests, Linear Pools use Linear Math. A linear pool will hold three tokens, including:
- two assets, i.e., main and wrapped tokens that have an equal value underlying token;
- the corresponding BPT (Balancer Pool Token). Note that BPT are ERC-20 tokens.
Nesting Linear Pools: Linear Pool BPT can be nested within another pool. This creates a simple batchSwap path between base assets and tokens in the outer pool because swappers can swap from BPT to one of the Linear Pool's underlying tokens.
Composable Stable Pools: Composable Stable Pools [3] are designed for assets that are expected to swap consistently at near parity or at a known exchange rate. Composable Stable Pools use Stable Math which allows for swaps of significant size before encountering substantial price impact, vastly increasing capital efficiency for like-kind and correlated-kind swaps.

A pool is composable when it allows swaps to and from its own LP token. Putting its LP token into other pools (or "nesting") allows easy batchSwap from nested pool tokens to tokens in the outer pool.
Boosted Pools: Boosted pools [4] are designed to improve the capital efficiency of idle liquidity for large pools. Boosted pools are actually a subclass of other pools. For example, a boosted pool could be built on top of linear pools.

Boosted Pools are designed to deliver high capital efficiency by enabling users to provide swap liquidity for common tokens while forwarding idle tokens to external protocols. This gives liquidity providers the benefits of protocols like Aave on top of the swap fees they collect from swaps.

0x1.2 A Concrete Example of the Vulnerable Boosted Pools: Balancer Boosted Aave USD

Balancer Boosted Aave USD (symbol: bb-a-USD) is a Composable Stable Pool that facilitates swaps between three stablecoins (i.e., USDC, USDT, and DAI) while sending idle liquidity to Aave. The underlying linear pools are:

bb-a-USDC (consisting of USDC and wrapped aUSDC)
bb-a-USDT (consisting of USDT and wrapped aUSDT)
bb-a-DAI (consisting of DAI and wrapped aDAI)

Specifically, bb-a-USD is a collection of one Composable Stable Pool that contains the pool tokens of three different linear pools, and each of those linear pools has an associated stable token: DAI, USDC, and USDT. The below figure provided by the official document [5] shows the structure of bb-a-USD :

0x1.3 How to Calculate BPT's Price

An important question that naturally arises is how to determine the price of BPT when swapping a specific amount (i.e., amountIn) of BPT for a certain amount (i.e., amountOut) of another token.

Balancer provides detailed description for the mathematic formulas they adopted [6, 7] by different pools. For the stake of simplicity, we abstract and summarize the most relevant concepts here.

Take the linear pool as an example, the BPT's price is calculated in the onSwap function of the LinearPool contract.

The calculation can be summarized as follows:

$$amountOut=amountIn*tokenRate$$

Here tokenRate is calculated with the following formula:

$_INITIAL_BPT_SUPPLY$ is a constant value: $2^{112} - 1$.

In the above formula, the numerator can be simplified as the sum of the main token's balance and the wrapped token's balance, while the denominator is the difference between a predefined value (i.e., _INITIAL_BPT_SUPPLY) and the balance of BPT.

It is worth noting that the balances of all involved tokens need to be nominalized before performing the calculation, because different tokens may have different decimals. Specifically, the raw balance of a given token will be multiplied with a corresponding upscale factor, which is determined by the _scalingFactors function.

(1) Scaling Factors of `Linear` Pools

Both BPT and the main token have a regular, constant scaling factor.

(2) Scaling Factors of `Boosted` Pools like `bb-a-USD`

The calculation of a boosted pool is a little bit complicated. Specifically, the returned scaling factor is the product of the raw scaling factor (e.g., 1e18) and the token rate, which is obtained from the cached token rate if any.

Where does the cached token rate come from? There exists a private function named _updateTokenRateCache function. Obviously, this function will first retrieve the rate by invoking the getRate function of that token and then caches it.

Again, take bb-a-USDC as an example, the core logic of the corresponding getRate function follows the formula we discussed earlier.

Note that there are three possible paths that can trigger the _updateTokenRateCache function:

Besides, there is an expiration check in place when performing updates for paths that via the onSwap function:

0x2 Vulnerability Analysis

The root cause lies in the price manipulation caused by the rounding down logic within the onSwap function of the linear pool. This, in turn, improperly affects the cached token rate used by the boosted pool.

Specifically, the amountOut is rounded down when the _downscaleDown function is invoked. Therefore, if there's a significant magnitude difference between amountOut and scalingFactors[indexOut], the return value of the _downscaleDown function could be zero.

For example, if we use bb-a-USDC (as BPT) to swap USDC (as the main token) in the bb-a-USDC pool, when amountOut is less than 1,000,000,000,000, the return value will always be rounded down to zero. This would increase the balance of bb-a-USDC as it could be regarded as adding liquidity of bb-a-USDC uni-directionally.

As a result, if BPT is the token used for the swap, its rate will rise in line with the formula to calculate the rate, given that the numerator remains the same while the denominator decreases. This bug could be exploited to lead to a (huge) price difference.

0x3 Attack Analysis

The attack transaction consists of the following attack steps:

Borrowing 300,000 USDC via Flashloan from Aave.
Swapping 1.067753 USDC for 0.970495 aUSDC in the bb-a-USDC pool.
Performing batchSwap in the bb-a-USDC and bb-a-USD pools, i.e., harvesting 15,628 bb-a-USDC, 139,431 bb-a-DAI and 248,868 bb-a-USDT with 42,203 USDC. The detailed steps are summarized in the following table (with decimals):

Swapping LP tokens for the corresponding underlying stable tokens:

139,431 bb-a-DAI -> 141,127 DAI in the bb-a-DAI pool
15,628 bb-a-USDC -> 15,685 USDC in the bb-a-USDC pool
248,868 bb-a-USDT -> 253,461 USDT in the bb-a-USDT pool

Repaying the flashloan, and the final profit is:

114,324 DAI
253,461 USDT
0.970495 aUSDC

It is worth noting that the attacker drained aUSDC with USDC from the bb-a-USDC pool in step 2, which would make the price manipulation in step 3 much easier, i.e., the attacker only needed to focus on USDC and bb-a-USDC.

Here step 3 plays the key role. Now let's delve into the details of this step to figure out why the attacker could make profits. Specifically,

Steps 3.1 is used to drain USDC with bb-a-USDC from the bb-a-USDC pool;
Steps 3.3 and 3.4 are used to swap bb-a-USDC for bb-a-DAI, while step 3.5 is used to swap bb-a-USDC for bb-a-USDT.
Step 3.7 is used to swap USDC for bb-a-USDC from the bb-a-USDC pool.

Here steps 3.2 and 3.6 do not swap back any target tokens (i.e., USDC) due to the rounding down discussed earlier, hence the balances of the target tokens remain unchanged after the swapping, which can be regarded as adding extra liquidity of bb-a-USDC into the bb-a-USDC pool.

Obviously, the abnormal swaps mainly occur in steps 3.4, 3.5 and 3.7. In the following, we will go through the details of each of these steps in turn.

(1) bb-a-USDC -> bb-a-DAI

In step 3.3, the exchange rate between bb-a-USDC and bb-a-DAI is almost 1, while in step 3.4, the exchange rate becomes 19:

Step 3.3: 1,000,339,378,515,783,699 / 1,000,000,000,000,000,000 = 1.00
Step 3.4: 139,430,482,942,020,211,267,110 / 7,300,000,000,000,000,000,000 = 19.10

Recalling the code logic we discussed earlier, in step 3.3, after returning the previously cached token rate to calculate the scaling factor (1,012,181,365,780,643,700), it updates the rate to calculate a new value (40,240,000,000,000,000,000). This updated value is then used in step 3.4 as the new scaling factor. Since the raw scaling factors remain unchanged (i.e., 1e18), this implies that the new rate is approximately 40 times greater than the old rate.

However, where does this significant increase originate from? Let's revisit the formula for calculating tokenRate. Since the balance of aUSDC has been depleted in step 2, the calculation of tokenRate can be simplified as follows:

Here the actual value of the nominalMainBalance is due to the rounding down occurs in step 3.2.

(2) bb-a-USDC -> bb-a-USDT

Step 3.5 uses the same trick to get more bb-a-USDT, and the exchange rate between bb-a-USDC and bb-a-USDT is more than 12:

248,868,905,733,352,246,491,156 / 20,000,000,000,000,000,000,000 = 12.44

(3) USDC -> bb-a-USDC

Furthermore, bptBalance is increased in step 3.6, then bptSupply becomes zero in step 3.7. By doing so, it is possible to swap USDC for bb-a-USDC at an exchange rate that is nearly 1:1.

0x4 Summary of Attacks and Profits

As of this writing, we have observed dozens of attacks in the wild, causing losses in excess of $2.12M. In summary, these attacks were executed by three distinct accounts, as follows:

Balancer suffered a total loss of ~$1M due to this vulnerability. Less than 12 hours after the initial attack on Balancer, its forked protocol, Beethoven X, succumbed to similar attacks, leading to an estimated loss of ~$1.1M. Beethoven X incurred even greater losses than Balancer! The cumulative loss from this security incident amounted to ~$2.12M.

A full list of these attack transactions has been collected in a document we prepared. Please refer to it for more detailed information.

Some Observations About the Attackers

Upon analyzing the transactions initiated by each network, we found a significant divergence in the trace of attack transactions on Fantom compared to those on Ethereum and Optimism.

Specifically, beyond the notable differences in key functions, the attacker on Fantom leveraged two unique tricks to avoid being front-run by MEV Bots. Furthermore, the funds used for the attack on Fantom were prepared 163 days in advance of the attack.

From the observations detailed above, we can infer:

At least two distinct attackers were involved.
The attacker on Fantom is an experienced serial offender.

0x5 Conclusion

In summary, this is a subtle vulnerability rooted in the rounding down logic. However, exploiting this vulnerability isn't straightforward. Specifically, the attacker was able to inflate the cached token rate by exploiting the rounding down issue in the linear pool, thereby manipulating the token price in the corresponding boosted pool.

This incident also emphasizes the importance of timely notification to those projects that have forked from the vulnerable source. Despite Balancer's alert, attacks aimed at forked protocols continue, highlighting the necessity for these forked projects to stay informed about security updates from their source projects. However, ensuring these forked projects receive prompt notifications presents an ongoing challenge for the community.

Furthermore, the continuous series of attacks underlines the importance of proactive threat prevention, which could effectively help mitigate potential losses.