[PROPOSAL 890] [PASSED] Signaling Proposal: IBC Rate Limiting

Hub Proposals Signaling/Text
1

TL;DR;

This proposal recommends a mechanism that may limit damage to user funds in the case of an exploited vulnerability. If it passes, the IBC Rate Limit module developed by Stride Labs will be added to the Cosmos Hub.

Overview

As the Interchain ecosystem and economy grows in importance, it becomes increasingly important to ensure that each sovereign chain and their inter-connections are protected against exploits.

There are a number of different ways to achieve this; exhaustive testing, code audits and a variety of defensive measures can also be enacted. No matter what measures are taken, there will always exist a potential for misuse in any complex system. Therefore, it is prudent to have defensive measures in place as well, especially because code bugs, environment and library vulnerabilities can manifest themselves in unforeseen ways. This proposal seeks the community’s opinion about the integration of a defensive mechanism to reduce the impact of any exploited vulnerabilities.

Vulnerabilities that exploit user funds can always be rolled back if an appropriate governance vote has taken place, as long as the sandbox is an isolated blockchain. If funds can be on-boarded or off-boarded to other chains, then the instant a vulnerability is exploited, funds can be moved to other chains, beyond the reach of any governance claw back from the exploited chain. The mechanism to off-board funds would be via any bridges to other chains or via IBC support. A rate limiting feature would limit the amount of incoming and outgoing traffic from a particular network that matches a specific criteria.

Previous Exploits and Other Implementations

Examples of previous real or potential exploits on the Cosmos Hub and other ecosystems are shown below:

The rate limiting technique is a useful tool that is implemented in a variety of scenarios, including the area of financial transactions, for example on the Osmosis chain.

Proposal

We propose adding the Rate Limit module developed by Stride Labs to the Cosmos Hub. The module prevents massive inflows or outflows of IBC tokens in a short time frame (e.g., 24 hour window). Every rate limit is applied at a ChannelID + Denom granularity. For example, a rate limit could be added for uatom on the Cosmos Hub <> Osmosis channel (channel-141).

Every rate limit will also have a configurable threshold that dictates the max inflow/outflow along the channel. The threshold is represented as a percentage of the total supply of the denom at the start of the time window, and it remains constant until the window expires. For instance, the rate limit for uatom on channel-141 might have a threshold of 5% for both inflow and outflow. Given the total supply of 388M ATOMs, such a rate limit would reject any IBC transfer that would cause a net inflow or outflow greater than 19.4M ATOMs. Once the time window expires, the net inflow and outflow are reset to 0.

Initially, we propose the add the following (conservative) rate limits (for both inflow and outflow) with a 24 hour time window:

  • 5% for uatom on Cosmos Hub <> Osmosis (channel-141) – a net flow of 19.4M ATOMs / day
  • 1% for uatom on Cosmos Hub <> Neutron (channel-569) – a net flow of 3.9M ATOMs / day
  • 1% for uatom on Cosmos Hub <> Stride (channel-391) – a net flow of 3.9M ATOMs / day
  • 1% for uatom on Cosmos Hub <> Kujira (channel-343) – a net flow of 3.9M ATOMs / day
  • 1% for uatom on Cosmos Hub <> Injective (channel-220) – a net flow of 3.9M ATOMs / day
  • 1% for uatom on Cosmos Hub <> Persistence (channel-190) – a net flow of 3.9M ATOMs / day
  • 1% for uatom on Cosmos Hub <> Secret (channel-235) – a net flow of 3.9M ATOMs / day

These limits are conservative enough to avoid false positives – user transfers being rejected – while still providing an initial protection against exploits. The proposed values are based on the following back-of-the-envelope calculation: In the last 14 days, ~400.000 ATOMs were transferred out of the Hub on average per day, which is ~ 0.1% of the total ATOM supply, and most of this is transferred to Osmosis (more than 90%). This makes the suggested 5% limit large enough to avoid false positives.

Once these rate limits are added, both the inflow and outflow on these channels can be monitored (gaiad q ratelimit list-rate-limits) and the limits can be adjusted accordingly. Note that the Rate Limit module enables rate limits to be added and updated via governance proposals.

Proposal Outcomes

The following items summarize the voting options and what they mean for this proposal:

Upon a YES vote:

  • The Rate Limit module developed by Stride Labs will be integrated into Gaia and rolled out in one of the next major releases.

Upon a NO vote:

  • The Rate Limit module will not be added to the Hub and the discussions with the community will continue to find the best path forward to introduce the rate limiting feature that helps to protect user funds.

NO WITH VETO - A ‘NoWithVeto’ vote indicates a proposal either (1) is deemed to be spam, i.e., irrelevant to Cosmos Hub, (2) disproportionately infringes on minority interests, or (3) violates or encourages violation of the rules of engagement as currently set out by Cosmos Hub governance. If the number of ‘NoWithVeto’ votes is greater than a third of total votes, the proposal is rejected and the deposits are burned

ABSTAIN - You wish to contribute to quorum but you formally decline to vote either for or against the proposal

References

7 Likes
2

Would there be a way to set the limits per channel via governance proposals, in case community would want to change these?

1 Like
3

Yes, this is why we’ve started with very conservative settings. We’d be happy to change these even in the initial signaling prop.

4 Likes
4

The proposed safety module design is super simple, and it’s an absolute necessity for the Hub’s defenses. Concerning the proposed limitation thresholds, we suggest slightly higher barriers for our esteemed AEZ partners, @Stride and @Neutron, enabling an increase in the 24-hour limit to 7.8M ATOMs/day.

We are also wondering if there are plans to include similar restrictions for other tokens currently held inside the Hub. Presently, we are holding 65,896,156, and it might be wise to consider implementing restrictions for these tokens as well.

image
image2808×640 81.1 KB

Source: Mintscan

Lastly, we wonder if the 24-hour cycle is pertinent, considering that top validators on the Hub operate advanced chain monitoring and would rapidly raise the alarm in the event of harmful activity being detected on our radars. We ask this question because we believe that a shorter life cycle could potentially allow us to lower barriers even more without disrupting regular users’ activity. For example, by setting the DurationHours parameter to 8 hours, we would effectively raise the allowed transfers four times a day with the same restriction thresholds.

If considered, we could even support lowering the MaxPercentRecv and MaxPercentSend to 4% for the Hub instead of 5%. This is the only improvement consideration we foresee.

To conclude, we are glad to see this proposal, and we will definitely support it on-chain with a warm YES vote! Thanks for reading,
pro-delegators-sign

5 Likes
5

@Govmos Thanks for the feedback.

Concerning the proposed limitation thresholds, we suggest slightly higher barriers for our esteemed AEZ partners, @Stride and @Neutron, enabling an increase in the 24-hour limit to 7.8M ATOMs/day.

We believe that the 1% limit is already very conservative – in the last 7 days, the Stride-Hub channel had a net flow of around 1.3M$ worth of tokens, while the Neutron-Hub channel had a net flow of around 1M$ (cf. Mintscan). Not sure how accurate this data is (it seems that different explorer show different data), but this is two orders of magnitude under the current limit of 3.9M ATOMs / day. Once the rate limit module is in production, we can closely monitor these channels and update the limits based on more accurate data.

Also keep in mind that increasing these per channel limits, would increase the overall limit of how much of the total ATOM supply can be transferred out of the Hub. With the current suggestion, that’s 11% per day. Given that ~63% is locked, this means that around of 1/3 of the total liquid ATOM supply could be transferred out of the Hub per day.

We are also wondering if there are plans to include similar restrictions for other tokens currently held inside the Hub. Presently, we are holding 65,896,156, and it might be wise to consider implementing restrictions for these tokens as well.

Adding a limit for NTRN on the Neutron-Hub channel is an interesting suggestion. We need to be careful though to not block any attempts on enabling the NTRNs in the CP to participate in Neutron’s governance – this is something we are working on enabling via the ICA controller module. We could whitelist the CP account to not be affected by this limit.

The main question here though is whether we want to limit the amount of NTRN flowing to/from Neutron. An alternative would be for Neutron to enable the Rate Limit module on their chain.

Lastly, we wonder if the 24-hour cycle is pertinent, considering that top validators on the Hub operate advanced chain monitoring and would rapidly raise the alarm in the event of harmful activity being detected on our radars. We ask this question because we believe that a shorter life cycle could potentially allow us to lower barriers even more without disrupting regular users’ activity. For example, by setting the DurationHours parameter to 8 hours, we would effectively raise the allowed transfers four times a day with the same restriction thresholds.

I agree that the configuration of these limits can be improved. However, given that it’s a new feature for the Hub, I think it’s better to be on the conservative side and allow validators to get used to the new requirements for monitoring.

If considered, we could even support lowering the MaxPercentRecv and MaxPercentSend to 4% for the Hub instead of 5%. This is the only improvement consideration we foresee.

I do believe that a few month of collecting data would give us the confidence to reduce the 5% limit to something much smaller, e.g., 1%.

1 Like
6

ou are absolutely correct in pointing that out. Our recommendation was based on the entire supply of ATOM, not taking into account the significant share of illiquid tokens. With this information in mind, we would like to retract our proposition, and the initial 5-1-1% seems appropriate. We will edit the original post to remove this part.

We have always advocated for a data-driven approach, and therefore, we fully align with that vision. Thanks for your valuable contribution to this discussion!

2 Likes
7

At first glance this seems like a good idea, and was just about to vote yes, but when I thought about it a bit more, and from the POV of a user of the infra (speaking for myself here and not Stakecito), it actually seems like a pretty bad idea that would:

  1. Inhibit liquidity, limiting the free flow of assets between chains and affecting arbitrage opportunities which are vital for market efficiency.

  2. Could result in transactions being denied due to exceeding the set thresholds, negatively impacting the user experience for folks transacting

I also question the effectiveness of rate limiting to address vulnerabilities. I don’t see how this adequately addresses the root causes of vulnerabilities, it could also still be circumvented by attackers.

Then there’s the question of governance overreach, with the community having to regularly vote on adjusting rate limits to keep up with evolving network dynamics, potentially leading to governance fatigue

2 Likes
8

At this time too in the market. Idk, just seems weird to me personally to impose rate limits. I understand the reasons why, just wondering if we should maybe wait for better solutions.

Edited to say: Aaah I see Osmosis has IBC rate limiting too (which I’ve never actually felt as a user). Still question this approach though. Earlier today I saw a tweet that said IBC isn’t audited nearly enough. Maybe instead of imposing restrictions on users, that and thorough testing should be done. We will vote yes on this though.

4 Likes
9

At Umee we implemented very efficient IBC rate limiting. It uses oracle data to do USD value based quota. However it’s easy to adapt it for % based quota.

4 Likes
10

@Winfred Thanks for the feedback.

from the POV of a user of the infra (speaking for myself here and not Stakecito), it actually seems like a pretty bad idea that would:

  1. Inhibit liquidity, limiting the free flow of assets between chains and affecting arbitrage opportunities which are vital for market efficiency.
  2. Could result in transactions being denied due to exceeding the set thresholds, negatively impacting the user experience for folks transacting

I agree that there is a risk that rate limits can affect user experience. That’s the reason we proposed very conservative limits for now. Also, we plan to monitor the net flow on these channels and update the limits accordingly. It’s very unlikely that the traffic between the Hub and Osmosis for example will increase 100x over night, so we don’t expect the UX to be affected.

Aaah I see Osmosis has IBC rate limiting too (which I’ve never actually felt as a user).

Exactly :slight_smile:

Then there’s the question of governance overreach, with the community having to regularly vote on adjusting rate limits to keep up with evolving network dynamics, potentially leading to governance fatigue

We do not believe that the rate limits need to be adjusted regularly. These limits can be seen as the last resort in case of an exploit. That’s the reason they are set to large values compared to the actual traffic. As long as there are no huge fluctuations in the IBC traffic, there is no need for governance intervention.

I also question the effectiveness of rate limiting to address vulnerabilities. I don’t see how this adequately addresses the root causes of vulnerabilities, it could also still be circumvented by attackers.

At this time too in the market. Idk, just seems weird to me personally to impose rate limits. I understand the reasons why, just wondering if we should maybe wait for better solutions.

Earlier today I saw a tweet that said IBC isn’t audited nearly enough. Maybe instead of imposing restrictions on users, that and thorough testing should be done.

IBC is being audited. If the amount of audits is enough or not, it’s a completely different question. I do believe that the Cosmos community can do more regarding the correctness assurance of the Cosmos stack. But no matter how much a code is being audited, reviewed and tested, there is no such thing as a 100% guarantee of correctness. That’s the reason we need additional defensive mechanisms to avoid the worst case scenarios.

2 Likes
11

Hi @robert.zaremba. Thanks for sharing. What’s the advantage of an USD value based quota over a % based quota? Also, what’s the advantage of the Umee design over the Osmosis design?

12

It’s all about the risk assessment. Do you want to measure it in USD value or in amount of tokens.

Our implementation has two fold quota:

  • per token in USD value. Currently $1.7m / 12h
  • total outflows in USD value. Currently $2.2m / 12h

Our quota window is also a param - we can increase or decrease it (eg from 12h to 24h).

Note, that total outflows quota (sum of token values) is impossible with the approach presented in OP (because how do you comapre 1atom outflow to 1usdc outflow?).

13

First of all, thank you for providing the rationale behind your concerns, this kind of feedback is vital to strike the best final decisions on any matter. You’re absolutely right to mention the risk this could pose to arbitrage opportunities which we expect to be a core functionality of ATOM in the greater Cosmos. Nevertheless, we have to remember that limiting the uATOM IBC flows on the Hub itself won’t affect the LSTs behavior. Afterall, they are the most suitable liquid form of ATOMs to perform such arbitrages. With this dual form we can ensure the best of both worlds. The capital (ATOM) remains safe and protected with the rate-limit, while the money (stATOM, stkATOM,…) can move freely in the whole Cosmos without those restrictions (unless LST providers decide to use rate-limits themselves).

2 Likes