Upcoming ICA bug fix (Cosmos Hub v8)
The Cosmos Hub team at Informal recently became aware of an issue on the Cosmos Hub, which we would like to fix as soon as possible. The issue involves Interchain Accounts (abbreviated as ICA), which allows other chains to create accounts on the Cosmos Hub. ICA is a part of the IBC library used by the Hub and many other chains. In the version of IBC currently running on the Hub, ICA is vulnerable to an issue which can cause newly generated ICA accounts to become inaccessible. This issue does not result in theft or permanent loss of funds.
On December 23rd, we became aware that Quicksilver (a liquid staking provider) had attempted to make an Interchain Account, and someone had exploited the bug to make the account inaccessible. Until the bug is fixed, Quicksilver cannot provide liquid staking services on the Cosmos Hub, and some ATOM in the account is inaccessible. No user funds have been affected.
Still, we are going to fix the bug as soon as possible. Over the past week we have been working on a Cosmos Hub release (v8) which fixes the ICA issue by upgrading the IBC version to 3.4. It also includes migrations to move the stuck funds out of the inaccessible account. When this release is running, it will allow ICA to work again.
Important: Until the Cosmos Hub is running v8 in production, new ICA accounts may become inaccessible when they are created. Existing accounts are unaffected.
How does this affect Replicated Security (Cosmos Hub v9)?
Hopefully this fix will have little impact on the Replicated Security timeline. We will be able to finish the v8 release which fixes ICA in parallel with the v9 release which introduces RS. Both codebases should be done by Friday, January 6th.
Starting on Monday, January 9th, v9 (RS) will need to go through an additional testnet and a final audit. This is because bugs were found during the audit and testnet late last year and the fixed code needs a final audit and testnet. These are both projected to take two weeks. During this time, we will have the opportunity to put the v8 (ICA fix) release through the release testnet, and create a governance proposal for it.
If all goes well, v8 (ICA fix) will be live in production shortly after the audit and persistent testnet of v9 (RS) wrap up.
TLDR
- There is an issue on the Cosmos Hub making it possible for an attacker to stop new Interchain Accounts from being created.
- No funds have been lost or are at risk.
- There will be a v8 release fixing this issue before the v9 release introducing Replicated Security.
- However, this should not delay Replicated Security significantly.
7 Likes
Thanks for a straightforward and informative post about the situation with ICA and Quicksilver. Glad to know this issue is being corrected, especially glad that user funds weren’t affected and aren’t at risk. I just have a couple of questions.
Do you mean 2 weeks each or 2 weeks for both?
Does this mean we will also need an another prop for ICS in the form of a v9? Which of v8 or v9 will be “Rho”?
1 Like
I don’t think there is any reason to block voting on the v9 upgrade on the completion of the V8 upgrade.
They can be voted on in parallel.
Probably the biggest constraint around timing is Chinese New Years on 22nd and the need to avoid doing upgrade that week.
Well, the main thing is that voting on v9 is blocked by remaining testnet and audit tasks. That’s why the v8 release won’t affect timelines much
Do you mean 2 weeks each or 2 weeks for both?
Two weeks for both but they will happen simultaneously
Does this mean we will also need an another prop for ICS in the form of a v9? Which of v8 or v9 will be “Rho”?
Yes, two props. v8 is Rho, and v9 is Lambda
1 Like
Update on timing:
v8 (ICA bug fix)
Due to some bugs in third-party modules discovered only once we had done a test upgrade from the current live Cosmos Hub state, we were not able to cut the v8 release on January 9th as we had hoped. Instead, it went out about a week later on January 16th. One learning from this is that we should have a nightly testing setup which attempts to upgrade the state of the current live Cosmos Hub to the current latest code. If we had this, we would have learned of the bugs months ago.
The upgrade will go live on the release testnet on Friday: Big Dipper | Testnet Block Explorer
Once the upgrade succeeds on the release testnet we will put the upgrade proposal on mainnet (probably on Monday or Tuesday).
v9 (replicated security)
We cut a final release of the RS module on Monday of this week. The audit of the RS module started this week, and will run for 2 weeks.
The release of Gaia v9 was delayed by about a week, as we were upgrading everything to IBC 4.2, from IBC 3.4. This is because 3.4 will have an end of life in March. We could have used 3.4, but that would have resulted in needing to release v10 very soon after v9 (which itself will be released very soon after v8), increasing error surface, hassle for validators, and stress on the team. This migration to IBC 4.2 is now done, and the release will be cut very early Monday morning, and go to the persistent RS testnet on Monday during the day. The persistent testnet will run for 2 weeks starting next week. The persistent testnet is there to test some final things specific to RS.
After the RS persistent testnet has tested everything to our satisfaction after 2 weeks (it will continue running after this, hence “persistent”), we will start the release testnet for v9. The upgrade from v8 to v9 in the release testnet will take 3 days, after which the voting period is 2 weeks.
All told, we are currently projecting a go live date of February 22nd for Gaia v9 with replicated security.
3 Likes
Update on v8 timing:
We intended to put up the governance proposal for the v8 release at the beginning of this week, on Monday the 23rd.
However, we noticed a strange issue where the full nodes in the release testnet stopped syncing. This issue has now been solved: Full node syncing issue · Issue #2073 · cosmos/gaia · GitHub
We also noticed that it appears that some REST endpoints have stopped working after the upgrade. These rest endpoints were accidentally deleted but they have now been brought back.
Due to these debugging efforts, the final v8 release is planned to be cut on Monday the 30th. I am not sure right now if another 3 day release testnet upgrade will need to be done, but likely it will not. If this is the case, the governance proposal for the v8 upgrade will go up on Monday or Tuesday.
v9 timing remains unaffected by this, although it is good to note that if these bugs had not been found in v8, they would have come up during the v9 release process.
2 Likes