[PROPOSAL #187][ACCEPTED] V9 Lambda upgrade (with Replicated Security)

I can’t figure out how to message on here lol, what’s your Twitter handle?

I saw your message on Telegram, will check my agenda in the coming days (a lot going on at the moment :P)

Hi team, thank you so much for working on the proposal and also for giving the community the opportunity to feedback it. We, at Kalpatech, have some questions that if answered would make us understand the proposal better.

Approval and decisions to launch a consumer chains

It is mention that each new consumer chain will need to be confirmed in a separate on chain governance proposal. Meaning that the one which is accounted for is individual staking voting power.
On the other hand, it is said that if more than ⅓ of the Cosmos Hub validators do not want to run a consumer chain, it will not run, regardless of how the governance vote went. Which means that in the end, it is the validators that are using the collective delegated stake to decide on which chains to validate and not the owner of the stake.
Are we able to make this more clearly in the content of the proposal?

Revenues

It would be great if we could add the token inflation distribution to Atom Holders as being a requirement and not just an option, or by case application fees if the chain does not have a token. Atom holders expect high revenues coming from the inflation portion of the consumer chains and would be great to not leave the possibility to opt out of it.

Governance

It is mentioned that for the consumer chain governance, Cosmos Hub validators will not need to participate in governance of any consumer chains, since they have their own governance mechanisms built. That means that the voting right of the validator is being entirely canceled and the validators will just need to implement the decisions of the token holders?
This situation might create some disagreements in the way decisions and operations are being carried on and implemented. As validators are highly invested in the success of the chain and are adding resources and bandwidth to operate it, while having no say in the outcome of the chain or the decisions which are made regarding its future implementations could be seen as difficult.

Validators running consumer chains

Any consumer chain needs 2/3 validators of the cosmoshub to start and operate the chain. Do all Cosmoshub validators need to join the consumer chain once it is live as to not risk downtime, or if they decide to not join a specific chain for validation, and that chain is live, they could do so, not validate the chain and they won’t be slashed for downtime?

Security problems- malicious slashing

It is said that one potential issue is the malicious slashing, in which a consumer chain sends instructions to the provider chain to unfairly slash validators. To prevent this, a “slash throttle” that limits the number of validators that can be slashed at once, reducing the potential harm to the Cosmos Hub’s consensus was included. But this measure does not prevent for a malicious consumer chain to still attempt to unfairly slash a few validators. Can the slashing instructions that are coming from the consumer chain be put in “quarantine” or some sort of alert system that requires a special signing or approval? Like freeze the state to which the instruction was sent as a record and meantime investigate and approve or deny the request.

Game of chains findings and audit

Can the findings of Game of Chains be made public and how those issues have been addressed? Also it would be great to analyze the audit reports and the recommendations which were highlighted there as well.
Can the Cosmos Hub community pool fund for an additional audit report of the RS tech? I believe, considering the high stakes of the implementation we want to make sure that we have receive a sufficient amount of feedback and review and in this case having independent audit companies to be considered as well would definitely help.

2 Likes

@Adriana will respond to all of these tomorrow, thanks!

3 Likes

Approval and decisions to launch a consumer chains
It is mention that each new consumer chain will need to be confirmed in a separate on chain governance proposal. Meaning that the one which is accounted for is individual staking voting power.
On the other hand, it is said that if more than ⅓ of the Cosmos Hub validators do not want to run a consumer chain, it will not run, regardless of how the governance vote went. Which means that in the end, it is the validators that are using the collective delegated stake to decide on which chains to validate and not the owner of the stake.
Are we able to make this more clearly in the content of the proposal?

This is just how Cosmos governance and consensus works. For example, if an upgrade proposal passes governance but then the validators do not want to run it, it won’t run. We already emphasized this dynamic more in this proposal than it is usually talked about on most proposals, leading some people to think that it’s actually something special about RS, which is not the case. So personally, I would prefer not to emphasize it even more.

Revenues
It would be great if we could add the token inflation distribution to Atom Holders as being a requirement and not just an option, or by case application fees if the chain does not have a token. Atom holders expect high revenues coming from the inflation portion of the consumer chains and would be great to not leave the possibility to opt out of it.

The RS code itself does not have any way to check what kind of payments will be made to the Cosmos Hub from the consumer chains. This is something that needs to be verified by people participating in governance during the consumer chain proposal voting period. So any requirement for inflation to be at a certain level would not be a code change, but a signaling proposal. Since this is an upgrade proposal dealing exclusively with code changes, I don’t believe it’s the right venue for signaling. A separate signaling proposal laying out inflation requirements might be better suited for this.

Also, in general I personally think that it should be decided individually for each consumer chain.

Governance
It is mentioned that for the consumer chain governance, Cosmos Hub validators will not need to participate in governance of any consumer chains, since they have their own governance mechanisms built. That means that the voting right of the validator is being entirely canceled and the validators will just need to implement the decisions of the token holders?
This situation might create some disagreements in the way decisions and operations are being carried on and implemented. As validators are highly invested in the success of the chain and are adding resources and bandwidth to operate it, while having no say in the outcome of the chain or the decisions which are made regarding its future implementations could be seen as difficult.

I think you are right that this is potentially challenging. One of our top priorities at the Informal and Hypha Cosmos Hub teams is supporting RS, and consumer chain upgrades are at the top of our list of things to facilitate and keep an eye on.

In the future, we will be upgrading RS so that malicious consumer chain code cannot harm the Hub in any way. Together with good practices around key assignment, and sandboxing of consumer chain binaries on validator hardware, this means that validators don’t actually have to care that much about the content of a consumer chain upgrade from a security perspective. This is a little bit like how Ethereum validators don’t care much about the content of smart contracts running there.

So, our goal for the future is that validators load up new consumer chain upgrades with doing much due diligence because no bad outcomes are possible, and things are as automated as possible. We are not quite there, but until then, we will be playing a very active role facilitating upgrades and preventing any issues.

Validators running consumer chains
Any consumer chain needs 2/3 validators of the cosmoshub to start and operate the chain. Do all Cosmoshub validators need to join the consumer chain once it is live as to not risk downtime, or if they decide to not join a specific chain for validation, and that chain is live, they could do so, not validate the chain and they won’t be slashed for downtime?

Here is a good summary of why this is not secure (the subset problem): Informal Blog - Replicated vs. Mesh Security

That being said, we will be setting the downtime slashing window very leniently on consumer chains. Probably around 4 days by default, and maybe much longer for the first few that are launched. This means that validators will have a very large margin of error around getting their hardware up and running on a consumer chain.

Security problems- malicious slashing
It is said that one potential issue is the malicious slashing, in which a consumer chain sends instructions to the provider chain to unfairly slash validators. To prevent this, a “slash throttle” that limits the number of validators that can be slashed at once, reducing the potential harm to the Cosmos Hub’s consensus was included. But this measure does not prevent for a malicious consumer chain to still attempt to unfairly slash a few validators. Can the slashing instructions that are coming from the consumer chain be put in “quarantine” or some sort of alert system that requires a special signing or approval? Like freeze the state to which the instruction was sent as a record and meantime investigate and approve or deny the request.

A consumer chain which contains malicious code that slashes a few validators unfairly will be removed immediately by the validator community and will not come back online. After this, governance proposals can be made to compensate the victims for slashes and time jailed, which will likely not be much money anyway.

Game of chains findings and audit
Can the findings of Game of Chains be made public and how those issues have been addressed? Also it would be great to analyze the audit reports and the recommendations which were highlighted there as well.
Can the Cosmos Hub community pool fund for an additional audit report of the RS tech? I believe, considering the high stakes of the implementation we want to make sure that we have receive a sufficient amount of feedback and review and in this case having independent audit companies to be considered as well would definitely help.

Simply Staking has a proposal up on the forum for a truly 3rd party audit. I support it. The Informal audit is still ongoing and will output a report in about 2 weeks. In the meantime, here are some things that they found that we have mostly already corrected: Issues · cosmos/interchain-security · GitHub

@lexa did we have some kind of public report or blog post on the outcomes of GoC?

1 Like

IMHO it should be discussed more because we’re talking about adding complexity and overhead to the validator operator’s operations. Beside the governance mechanics being constructed similarly, a validator operator supporting an upgrade and a validator operator spinning up additional infrastructure and increasing overhead to support a new chain is not an apples to apples comparison.

In a bear market, like the one we’re in right now, economic and financial feasibility of supporting numerous consumer chains should be given greater consideration. We are no longer operating from a place of abundance.

It’s not clear that incentives between delegators and delegates are aligned. ATOM stakers are incentivized to add consumer chains and potentially earn additional rewards. Many will choose the lowest commission validator. It’s unclear whether validator operators’ support of consumer chains will be profitable (or whether delegators even care).

1 Like

Hey everyone,

Trying to figure out how much ICS costs would actually be for new consumer chains, and would appreciate any clarifications you’re able to give re my back-of-the-envelope calculations below.

A figure I’ve heard being thrown around is roughly 600-800 dollars per month for validators, times 175 validators. In addition to that, on top of the rewards coming from consumer chains the validators get an average fee of around 10% (meaning we’ll have to multiply the results by 10 to get the revenue needed to cover such costs).

The result would look something like this:
$600 x 175 validators x 12 months x 10 = $12.6 million per year
$800 x 175 x 12 x 10 = $16.8 million per year

Although this does not consider the weight of different validators, these figures are quite extraordinary, and surely cannot be correct? Hard to see new chains being able to cover this kind of amount, especially so during a bear market like this. Or am I missing something?

Further, I’m wondering whether or not there is a normal investment risk for validators and the Hub (so if the project goes badly, the validators just lose money and vice versa) or if there are any requirements about what a consumer chain should provide in terms of value to the Hub in order to join Replicated Security?

Hope that makes sense, and again, I’m grateful for any light that can be shed on this matter.

3 Likes

Great post and excited about Replicated Security! What is the current timeline for this to be submitted as an on-chain vote/how long does this current consultation period last?

1 Like

Hi, thank you for the post.
I remember that GoC, unfortunately, ended without testing the slash feature, because the provider chain stopped as soon as the Slasher chain started sending slash packets. Was there any update on this after GoC?

@jtremback and @Adriana - we are working on putting together a report on identified bugs and how they’re being fixed (or have been fixed) for the v9 mainnet release. I’ll link it here when we’re done.

@BIN

Correct - GoC ended before this feature was fully tested. The slash rate throttling will be the first thing we test on the persistent testnet (aiming to launch next week).

We won’t be moving on-chain for mainnet until it’s tested and resolved.

@LanreIge1

We need to get v8 out first, and that will take at least two weeks (voting period!) from when it goes on chain. We also need to finish testing the slash rate throttle, as I mentioned. At the very earliest, we could be on-chain the week of February 6.

3 Likes

That makes sense, thanks for the response and all the work.

1 Like

Is it 2/3 of the validator set only or staked ATOM?

Voting power based. So 2/3 of the total voting power (thus staked ATOM)

1 Like

(post deleted by author)

Thanks!
How long is the voting period for such proposal?
And how long it the “preparation” period, so validators can set up the consumer chain nodes?

Same parameters as other governance proposals - 2 week voting period.

The ‘preparation period’ lasts as long as it takes for 2/3 of the set to bring nodes online, since there is no penalty until the chain is live. To the best of my knowledge, there’s no time-based parameter for how long validators have to come online.

There is a “spawn time” in the replicated security proposal. The consumer chain can start after this time passes. So an extra preparation time could be built in by setting the spawn time to some time after the end of the voting period.

1 Like

Hi everyone, we’ve decided to update RS to make some forms of slashing for offenses on consumer chains less automated in this first release to ensure the safety of the Hub. You can read about it here: Slashing updates in replicated security

This should not have any effect on the safety or liveness of consumer chains.

6 Likes

@jtremback
I thought this upgrade was supposed to include the Liquid Staking Module. Was the LSM cut from the roadmap or pushed down the road?

Can anyone tell me why Inter-chain Security was changed to Replicated security? The name may be a better descriptor of the mechanism, however the change causes confusion? Who signed off on the name change and how was it justified?