[PROPOSAL #187][ACCEPTED] V9 Lambda upgrade (with Replicated Security)

V9 Lambda upgrade (with Replicated Security)

Terminology note: Interchain Security is the name for a family of protocols. The feature being discussed here is in this family, but more accurately termed “Replicated Security”. In the past it has been referred to as Interchain Security v1, or ICS, but going forward we refer to it as Replicated Security.

Changes to this proposal

This proposal went onto the governance forum in mid-December. Since that time, we have made one change to the protocol. In response to the only high-severity finding in the Informal audit report, and feedback from the community, we have made it so that the slashing and tombstoning of validators for offenses on consumer chains must be approved by a governance vote. This will have a very minimal impact on the operation and security of consumer chains while protecting the Cosmos Hub from possibility of an attack by a rogue consumer chain.

This governance gating is temporary until the Cosmos Hub is able to cryptographically verify proof of double signing in an upcoming release of Replicated Security. This does not give governance the ability to slash validators arbitrarily, but rather to approve slash packets that have already been transmitted from a consumer chain.

Summary

The Cosmos Hub is upgrading its security system with a new feature called Replicated Security. This will allow the Cosmos Hub to provide its strong security to other blockchains, which are called “consumer chains”. The cost to censor or control a consumer chain with an economic attack is the same as the cost to censor or control the Cosmos Hub itself. This means that consumer chains can benefit from the Cosmos Hub’s security without having to maintain their own validator sets.

Replicated Security works through the IBC (Inter-Blockchain Communication) protocol. Consumer chains receive periodic IBC packets containing the up-to-date validator set of the Cosmos Hub. They use this information to update their own validator sets, effectively replicating the Cosmos Hub’s validator set across all consumer chains. This means that the Cosmos Hub’s validators can validate multiple blockchains at once with the same stake.

In exchange for this service, consumer chains are expected to send a portion of their fees and inflation to the Cosmos Hub validators and delegators. These tokens will then be included as part of the Cosmos Hub’s staking rewards.

The ATOM economic zone

It is anticipated that consumer chains using Replicated Security will collaborate closely in an “ATOM economic zone”. This is expected to have a snowball effect, with the Cosmos Hub securing a growing group of high-value decentralized protocols. Already, some prominent projects such as a smart-contracting platform, an AMM, several liquid staking providers and others have expressed interest in using Replicated Security from the Cosmos Hub.

Replicated Security can also be used to further the cause of “Hub minimalism”. Any new features to the Hub, even if they are core to the Hub’s functionality, can be launched as consumer chains. This allows the Hub to scale better and separate release cycles, enabling faster deployment of new features.

Onboarding a consumer chain

This governance proposal will not launch any consumer chains on its own. Instead, it will only add the necessary code to the Cosmos Hub to enable Replicated Security. In order to launch, each consumer chain must submit their own governance proposal to be voted on separately.

It is expected that before launch, a consumer chain will have participated in a Cosmos Hub Replicated Security testnet, conducted their own testnets, and undergone an audit by reputable auditors.

Once a consumer chain proposal passes, Cosmos Hub validators can begin running the consumer chain and receiving rewards. The chain will only start if more than ⅔ (by power) of the Cosmos Hub validators decide to run the consumer chain.

Offboarding a consumer chain

Once the consumer chain is running, validators can be slashed and jailed for consensus faults such as downtime and double signing on the consumer chain.

At any point in time, ⅓ (by power) of the Cosmos Hub validators will be able to stop running the consumer chain at once: the chain will halt, and none of them will be slashed for downtime.

The offboardings of consumer chains will have strong social norms to protect both sides, and any kind of forceful exit will only be coordinated in extreme cases such as protocol failures or attacks.

For normal cases, a separate governance proposal will be made to remove a consumer chain, which will allow Cosmos Hub validators to stop running it.

FAQs

How will the community ensure the quality of the consumer chains?

First, this proposal by itself does not start any consumer chains. Each consumer chain will need to be confirmed in a separate proposal. In other words, if you support Replicated Security but only want consumer chains that meet very high standards, it would be appropriate to use your “No” vote for the proposals of consumer chains that do not meet this standard.

Second, if more than ⅓ of the Cosmos Hub validators do not want to run a consumer chain, it will not run, regardless of how the governance vote went. This applies both to the start of the consumer chain, and continues to apply throughout its life.

For the avoidance of doubt, no validator will ever have to run, or will be slashed by a consumer chain which has not first been approved by governance, and then “accepted” by ⅔ of the validators intentionally choosing to run the consumer chain code.

Why would a Cosmos Hub validator want to run a consumer chain?

Replicated Security is a long awaited feature that will put the Cosmos Hub at the center of a constellation of valuable ATOM-aligned consumer chains. The consumer chains who will be introducing proposals to launch over the next year are some of the most highly-anticipated projects in the Cosmos space.

Replicated Security incorporates some fee-split mechanisms where a percentage of the value-captured by each consumer chain is sent to the Cosmos Hub validators and delegators.

Over time the fees may be raised as the value of being in the exclusive set of Cosmos Hub consumer chains is recognized, or perhaps lowered as validators improve their infrastructure and practices to be able to spin up new consumer chains more cheaply.

Over the long-term, Cosmos Hub validators and delegators may also consider a possible appreciation of their staking rewards as replicated security establishes itself as an important value-capture mechanism for ATOM.

In any case, governance will need to look at each consumer chain separately and decide whether the fee split makes sense for the consumer chain in question.

How should validators think about revenues and costs?

There are multiple ways a consumer chain could remunerate the Cosmos Hub validators

  1. Transaction fees: By default, 25% of the consumer chain fees will be distributed to the provider chain. This number can be set differently for each consumer chain. Note that validators also have control over the minimum fee they are willing to receive.

  2. Token inflation: If the consumer chain has a token, they can allocate a portion of the inflation to the Cosmos Hub validators and delegators (in other words, “continuous airdrops”).

  3. Application fees: If the consumer chain doesn’t have a token, a percentage of the swap fees, usage fees etc. can be distributed to the provider chain as well.

Each consumer chain will be responsible for putting up a governance proposal outlining a commitment to use a combination of these revenue streams to compensate the Cosmos Hub community for the replicated security service.

On the expense side, the hardware costs of running a new chain may range from $100 to $1000 based on the chain requirements and the technical setup selected by the operators. Informal Systems will be working with validators during 2023 to analyze resource usage and recommend cost-effective validator setups.

Another important component of the expense of running a chain is staying on top of upgrades and governance. For consumer chains upgrades, Hypha Coop and Informal Systems will establish a regular and predictable cadence of upgrades to each consumer chain to minimize time spent by validators. For consumer chain governance, Cosmos Hub validators will not need to participate in governance of any consumer chains, since they have their own governance mechanisms built.

Can the validators stop validating a consumer chain?

If it turns out that a given consumer chain is no longer worth validating because it is unreasonably taxing on validator infrastructure, or because it does not have high potential for future growth, then it can be stopped by a governance proposal.

In the worst case scenario, such a low-performing consumer chain can be stopped by ⅓+ of the validators simply turning off their nodes. As discussed above, in this scenario, no validator will be slashed for downtime.

What security problems could a malicious consumer chain cause?

The protocol design of Replicated Security currently assumes that consumer chains are audited before they are approved for launch. Reputable independent auditors are expected to conduct these audits. Before approving any consumer chains, governance voters are encouraged to review the audit findings. Auditors of consumer chains are encouraged to include a section in their reports that details the code’s adherence to the correct Replicated Security protocol.

However, we have also taken the following steps to limit the damage that a malicious consumer chain could cause in several scenarios.

One potential issue is malicious slashing, where a consumer chain attempts to slash every validator on every validator on the Hub at once by falsely sending packets claiming that the validators equivocated (double signed) on the consumer chain. To prevent this, slashing for equivocation on consumer chains must additionally go through a governance process before the validator is actually slashed. This is an extra layer of safety, and will have very minimal effects on consumer chains (the only concrete effect is that they will need to set an IBC parameter known as the “trusting period” to 5 days). In the next version of Replicated Security, the Cosmos Hub will be able to verify equivocation evidence automatically, and we can remove the governance gating.

Another potential issue is malicious jailing, where a consumer chain attempts to jail every validator on the Hub at once by falsely sending packets claiming that the validators had downtime on the consumer chain. However, these jailing packets are throttled, so that only one percent of the Hub validator power can be jailed per hour. Combined with the fact that jailing only lasts 10 minutes, it will not be possible for a malicious consumer chain to cause anything worse than transient disruption.

It should also be noted that the parameters around downtime jailing (e.g. how long a validator can be down on a consumer chain before being slashed) cannot be modified by a consumer chain’s governance. These parameters will be configured to be more lenient on consumer chains than on the Cosmos Hub.

Another potential issue is the misuse of validator keys. Validators can choose to use the same private key on both the provider and consumer chains, but this exposes them to the risk of their key being misused by a malicious consumer chain. To avoid this, validators can assign a different key for each consumer chain, which will prevent any potential misuse of their key.

Finally, the ability for Cosmos blockchains, including consumer chains, to use fully custom binaries could potentially be used to harm the underlying infrastructure. For example, a Cosmos chain binary could hypothetically erase the hard drive it was on. However, validators use virtualization technologies that allow them to run a different virtual machine for each chain, including consumer chains. This will prevent any harm to the underlying hardware.

Overall, while malicious consumer chains pose security risks, we have implemented measures to limit the potential damage they could cause.

How was the correctness of the Replicated Security ensured?

The Cosmos Hub team at Informal Systems used a wide variety of approaches to ensure the correctness and security of the Replicated Security module. It was then audited by the Model Based Testing team at Informal, as well as several other teams.

Confidence in the correctness was increased of the ICS code through various testing approaches:

  • Unit tests are useful for simple standalone functionality, and CRUD operations.

  • End to End tests utilize the IBC Testing Package, and test functionality that is wider in scope than a unit test, but still able to be validated in-memory, i.e., code where advancing blocks would be useful, simulated handshakes, simulated packet relays, etc.

  • Differential testing is similar to e2e tests, but they compare the system state to an expected state generated from a model implementation.

  • Integration tests run true consumer and provider chain binaries within a docker container and are relevant to the highest level of functionality. Integration tests use queries/transactions invoked from CLI to drive and validate the code.

Worst-case scenarios circuit breakers were added to ensure the stability of the Cosmos Hub:

  • Governance gating of slashing – Makes it so that slashing a validator for equivocation (double signing and related attacks) on a consumer chain must be ratified by Hub governance. As a result, an attacker that compromises a consumer chain cannot jail all the Cosmos Hub active validators at once and replace them with its own validator set, since the obvious fraudulent slashes will be voted down.

  • Jail packet throttling – makes it impossible to jail more than 1% of the power on the Cosmos Hub per hour. For more details, see github issue.

  • Unbonding liveness – makes it impossible for un-delegations to never complete (which would result in users never getting their staked Atoms back).

Several audits were completed with multiple ecosystem partners:

  • Protocol audit performed by the Apalache team at Informal Systems

  • IBC audit performed by the ibc-go team at Interchain

  • Informal Systems Audits team has provided continuous security feedback, and has conducted an audit

An incentivized testnet was also run for 4-5 weeks (Game of Chains) and findings were analyzed.

31 Likes

This is a no-brainer yes :heart:

7 Likes

Excited about this!

Do you foresee other challenges for validators besides the ones explained here?

Also, probably dumb questions but is there a limit on consumer chains the hub can secure?

2 Likes

I think we’ve got everything on here. We’re always looking for more feedback, but the main concern we’ve heard is infrastructure cost and time spent.

We’ll be working with validators to reduce infrastructure cost of consumer chains while maintaining the same security and performance, and we’ll be working with consumer chain teams to orchestrate chain upgrades in a steady and predictable cycle to reduce time spent.

3 Likes

It’s basically permissioned. Which is a good thing…

ICS will bring so much value to cosmos, the validators are going to a do a good job.

It’s no easy task, I can see it will be a big job to monitor all of this.

I say YES… I can’t wait to see what else follows and these chains that pay fees to validators in return will be paid out to stakers I assume.

This is a GAME CHANGER!

2 Likes

Good proposal.

For now, that’s exactly our concern as a validator. Hope we can have some sustainable solutions to this. Thank you.

1 Like

Thanks for yours and Informal’s work on ICS and thanks for putting up this draft proposal Jehan.

2 Likes

It seems very good balance in here; We look forward for other people’s feedback, but our early assumption is that this seems well crafted, with enough room to adjust on a case to case basis without introducing too much complexity in return. :+1:

3 Likes

Thank you for explaining it so well. It looks like a well-balanced proposal. But I have a few questions.

1- Can a consumer chain cause a provider chain halt in this case the Hub? if yes how can we avoid that?
2- When you say 2/3rd of validator needs to say yes it means 117 validators currently or the 2/3 of total val turnout. Because there are a few dozen Val who doesn’t participate at all.
3- Validator funding should be more explained as not all of the validators are well-funded.

Thanks in advance

2 Likes

2/3 refers to 2/3 of the total staked ATOM on the Hub.

ATOMs are staked with validators, which is why we sometimes refer to it as “2/3 of validators” when it’s shorthand for “validators who cumulatively hold 2/3 of the total stake”.

Enabling a consumer chain requires a YES vote by governance but it also requires validators to set up a node and run a new binary - there may very well be validators who will do that work but still not participate in tokenholder governance.

3 Likes

Yes, it totally makes sense, idk from where i got that.
16.88 VP of 23 validators never participated in the Governance. this means roughly 80% from ~83% of the remaining vp from 152 validators must vote yes to pass the prop. this is ambitious.

2 Likes

Are you talking about governance voting or having validators choose to run the code? The quorum and passing threshold for governance votes is the same for this proposal as it is for others, so no more ambitious than any other proposal. The Hub docs were recently updated to clarify some of these governance voting concerns, so you might find them helpful: On-Chain Proposal Process | Cosmos Hub

For governance votes, 40% of stake must participate (voting yes, no, nwv, or abstain) to reach quorum. Once quorum is reached, if >50% of the participating voting power (i.e., All the stake that has cast a vote in this particular proposal) has voted YES and <33.4% has voted NWV, the proposal passes.

The 2/3 of voting power refers to having validators actually run the code, which is the same requirement as running the Hub itself! Like I said before, validators who don’t participate in governance are still participating in securing the chain.

3 Likes

Simply put.

Simple gov prop to onboard code > Val will start the node but the chain will only start when 2/3 VP reaches > once the chain starts any active hub validator not running any of the consumer chains will be slashed (downtime).

1 Like

*.— [Unbonding liveness ]– makes it impossible for un-delegations to never complete (which would result in users never getting their staked Atoms back). -----
This is terrific, means delegators can lose their staked assets? could you please give us a deeper explanation of when and why this could happen?
Thanks.

1 Like

Simple gov prop to onboard code > Val will start the node but the chain will only start when 2/3 VP reaches > once the chain starts any inactive hub validator not running any of the consumer chains will be slashed (downtime).

Just a few clarifications, as I want to be sure we’re on the same page about validator activity and running “any” of the consumer chains.

The following will have to happen anytime a consumer chain is proposed:

  1. Simple gov prop on the Hub to onboard code, requires normal gov quorum and ‘YES’ threshold to pass.
  2. Once passed, validators will start their nodes and the specific consumer chain will only start when 2/3 of the Hub’s voting power is running the chain code.
  3. Once the chain starts, any active Hub validator not running the specific consumer chain will be slashed for downtime.

Inactive Hub validators do not participate in producing blocks for the Hub or any consumer chain. and are thus not at risk of being slashed for downtime until they join the active set.

1 Like

@zero_00

[Unbonding liveness ] - makes it impossible for un-delegations to never complete (which would result in users never getting their staked Atoms back).

This is saying that there was a worst-case scenario in which an undelegation message could be frozen and thus never complete, resulting in users being unable to unstake their ATOMS. Bad stuff, for sure.

The good news is that this is was resolved - the proposal text here says that the ‘unbonding liveness’ circuit breaker was added to make such a situation impossible in the future.

I can see why the double negative is confusing! Maybe best to read it as:

[Unbonding liveness ] - Fixes an issue in which an undelegation fails to complete, resulting in users never getting their staked ATOM back.

Related Github issues as I was looking into this:

6 Likes

Thank you, there was a typo, but this is same what I said.

This leaves us with the validator funding, as mentioned it costs 100-1000 to operate a node. for the sake of argument, we say the cost is $450 for a node, at the current price it is 50 Atom.

A validator with 200K delegations and 3% commission earns ~100 Atom monthly. bottom 75 validators have less than 200K atoms staked.

How can a validator run multipole consumer chains with this maths? and theoretically, they are dependent on the consumer chain tokenomics for this funding.

1 Like

I would like this part changed. Having the same set of validators potentially securing lots of consumer chains is still centralized. Decentralisation should only be used if it meets 3 characteristics:

  • decentralisation in location (global spread)
  • decentralisation in voting power (as @waqarmmirza rightfully indicates, we have a serious issue here on costs to operate validators in the lower ranks)
  • decentralisation in validators

Replicated security is a serious risk imo regarding validator decentralisation; which is something we should take seriously. The barrier for new validators to join the active set will only become bigger over time. And bigger validators have more people to run the stuff, so feeling more secure for a lot of people; probably causing more centralisation of voting power. How are these things tackled?

1 Like

We can certainly take the word “decentralized” out of the proposal if a lot of people object. It’s a term that is impossible to define rigorously. For example, many Bitcoin maximalists believe that no proof of stake chain can ever be decentralized.

What we can say for certain is that all consumer chains will be just as decentralized as the Cosmos Hub. Whether or not the Cosmos Hub is currently decentralized is outside of the scope of this discussion. I think it is.

The question of whether replicated security will make the Cosmos Hub less decentralized by hurting small validators is completely valid though. This is the main concern we have heard, and it will be the Cosmos Hub team’s top priority for 2023. We will be making sure that consumer chain testnets and upgrades are as smooth and regular as possible to minimize the human workload on validators. We’ll also be working with validators to review the infrastructure cost breakdown of their setups and see how we can implement code to lower it for consumer chains.

Finally, this proposal by itself does not launch any consumer chains, it just installs the code to make replicated security possible. The question of whether a given consumer chain is worth running is best asked when that consumer chain is being proposed.

6 Likes

Thanks a lot for the fast and precise reply-

1 Like