[PROPOSAL 854][PASSED] Gaia v14.1 Upgrade

[btruax]

[PROPOSAL 854][PASSED] Gaia v14.1 Upgrade

Change log:

• 2023-11-08 Created initial post
• 2023-11-15 Clarified FAQ on retroactive tombstoning
• 2023-11-17 Updated text to reflect v14.1 and outlined reasoning for changes
• 2023-11-28 Updated title for on-chain proposal
• 2023-11-30 Updated to add link to release

An Important Note for Validators

tl;dr

• Never reuse keys between chains, including testnet chains.
• If you are not sure where you have used your current keys, do a key reassignment ASAP.
  • Here are the docs on how to do a reassignment.

For the purposes of this post, we’re referring to double signing as when there are two signatures from the same key on the same block height, on the same chain-id. “Equivocation” covers other misbehaviour but there has been very little confusion about how the cryptographic equivocation applies to anything but double-signing.

In order for double-signing on a consumer chain to be punished by the Hub, all of following must be true:

• Same chain-id
• Same block height
• Same keys on both signatures

If any of the above are not true, there will be no automatic punishment via the cryptographic equivocation feature.

New feature: Height-base filter for consumer equivocation evidence

There have been concerns about multiple valid chains with the same chain-id, such as prior testnets, on which validators may have used their mainnet keys.

A feature has been added that does the following:

• For new consumer chains:
  • Records the consumer chain block height at the time it enters the Hub’s security.
  • Filters out equivocation evidence that occurred prior to that block height.
• For existing consumer chains:
  • Records the consumer chain block height at a specified time:
    • Neutron: block height 4552189
    • Stride: block height 6375035
  • Filters out equivocation evidence that occurred prior to that block height.

In effect, this means equivocation on consumer chains prior to entering the Hub’s security will not be punished by the Hub. For existing consumer chains which are already within the Hub’s security, any equivocation prior to today will not be punished.

“automatic” refers to evidence being recognized and presented by Hermes relayers running in ‘evidence mode’, which is recommended to all Hermes operators after the 14.1 upgrade passes. Evidence can also be manually prepared, though only if the double-signing genuinely occurred on-chain (i.e., one cannot fabricate a double-signing incident).

Background

The Gaia v14.1 release is a major release that will follow the standard governance process by initially submitting this post on the Cosmos Hub forum. After collecting forum feedback (~ 1 week) and adapting the proposal as required, a governance proposal will be sent to the Cosmos Hub for voting. The on-chain voting period typically lasts 2 weeks.

On governance vote approval, validators will be required to update the Cosmos Hub binary at the halt-height specified in the on-chain proposal.

Proposed Release Contents

The relevant Github epic for this release is: Gaia v14.1.0 Release

The changes planned for the v14.1 release involves the update of the Interchain Security provider module for the Cosmos Hub. This update introduces a new feature: the cryptographic verification of equivocation. This feature will automatically punish validators for equivocation by verifying evidence of double signing and light client attacks on the provider chain.

Consequently, this change renders equivocation proposals, exemplified by proposal 818, unnecessary. After the upgrade height, the v14.1 release marks the deprecation of equivocation proposals.

The relevant forum post for the cryptographic verification of equivocation feature can be found here.

Testing and Testnets

The v14.1 release will go through rigorous testing, including e2e tests, integration tests, and differential tests. Differential tests are similar to integration tests, but they compare the system state to an expected state generated from a model implementation. In addition, v14.1 will be independently tested by the team at Hypha Co-op.

Validators and node operators can join a public testnet to participate in a test upgrade to a release candidate before the Cosmos Hub upgrades to the final release. You can find the relevant information (genesis file, peers, etc.) to join the release testnet (theta-testnet-001) here, or the Replicated Security testnet (provider) here.

Potential risk factors

Although very extensive testing and simulation will have taken place there always exists a risk that the Cosmos Hub might experience problems due to potential bugs or errors from the new features. In the case of serious problems, validators should stop operating the network immediately.

Coordination with validators will happen in the #cosmos-hub-validators-verified channel of the Cosmos Network Discord to create and execute a contingency plan. Likely this will be an emergency release with fixes or the recommendation to consider the upgrade aborted and revert back to the previous release of Gaia (v13).

Governance votes

The following items summarize the voting options and what it means for this proposal:

YES - You agree that the Cosmos Hub should be updated with this release.

NO - You disagree that the Cosmos Hub should be updated with this release.

NO WITH VETO - A ‘NoWithVeto’ vote indicates a proposal either (1) is deemed to be spam, i.e., irrelevant to Cosmos Hub, (2) disproportionately infringes on minority interests, or (3) violates or encourages violation of the rules of engagement as currently set out by Cosmos Hub governance. If the number of ‘NoWithVeto’ votes is greater than a third of total votes, the proposal is rejected and the deposits are burned.

ABSTAIN - You wish to contribute to the quorum but you formally decline to vote either for or against the proposal.

Frequently Asked Questions

What should I do if I currently use my provider keys on consumer chains?

• Perform a key assignment so that you are not reusing keys between chains.

How is consumer chain equivocation currently handled on the Hub?

• Evidence of the equivocation must be manually gathered and submitted along with an equivocation proposal via governance.

How will consumer chain equivocation be handled if it occurs between now and the v14.1 upgrade height?

• The equivocation proposal process could be used if it is completed before the upgrade height.
• Or the evidence could be presented after the upgrade height and the below process would apply.

How will consumer chain equivocation be handled after the upgrade?

• Equivocation proposals no longer exist.
• Once this upgrade passes, relayer operators running Hermes between Hub and consumer chains should upgrade to latest version.
• Evidence must be presented (either by running Hermes in evidence mode or manually formatting the message) and then slashing will happen automatically.

Can validators be retroactively slashed for past equivocation on consumer chains?

• No. Any equivocation from before the specified block heights cannot be punished.
• This change was made to ensure that there will be no slashing or tombstoning for double-signing incidents that have already been judged (e.g., Prop 818).

What will happen specifically to Pupmos and Citizen Cosmos?

• See answer above.

How does key reassignment interact with this feature?

• After key assignment, the previous key is only held in state for three weeks (21 days). While the old key is in use or held in state, any equivocation that occurred using that key can be punished. During this 21 day period, any equivocation using the old key can still be punished. Once 21 days pass and the old key is no longer held in state, equivocation using that key cannot be punished.

[jagachu]

Hello
The proposal to adjust the validator commission by a minimum of 5% has been approved. Has it been adjusted in conjunction with this upgrade?

[lexa]

Prop #826 Set Minimum Commission to 5% has spawned a GH issue for implementing this feature: Minimum Commission for proposal 826 · Issue #2768 · cosmos/gaia · GitHub.

It won’t be in v14, but looks like it might go into v15, which is scheduled for 2 months from now.

[mpoke]

More context on the decision to push the Minimum Commission feature to v15

Due to the timeline of the v14 release (a proposal is expected to be submitted on the Hub by Nov 17), this feature will go into Gaia v15. Since we want to avoid releases around the winter holidays, v15 is scheduled for January 2024. Also, v15 will most likely update SDK to 0.47, which enables setting the minimum commission through a gov param. Therefore, in v15 we’ll just set the default of this param to 5%.

Note though that just setting the param to 5% will not change the commission of existing validators. The param affects only the MsgEditValidator and MsgCreateValidator messages. To enforce the existing validators to have a minimum commission, migration code needs to be written. This code would enforce all validators to have at least 5% commission.

[Cosmic_Validator]

@btruax

As testnet coordinators going forward, we will work with incoming chains to make sure their testnet chains always have ‘test’ (e.g., ‘consumername-test-1’) in the name (which is unlikely to be reused as a name) but this was not always the case so it remains a risk for any older chains.

So far there are only two consumer chains, Neutron and Stride. @lexa @jtremback you don’t have the information about all testnet names for Neutron and Stride before joining as consumer chains? With this information there can be certainty whether some old Neutron or Stride testnets had the same chain-id and whether there is risk about this or not, and if there was indeed some old testnet with same chain-id then all validators should be informed/reminded about this immediately.

@btruax

If your validator has ever double-signed with the same keys on that consumer chain, it is at risk of being tombstoned after the upgrade.

Does this only affect @serejandmyself and @pupmos? Because so far there was only one consumer chain double sign equivocation proposal on the Cosmos Hub.

@btruax

So validators could get retroactively tombstoned for equivocation that happened ages ago?

• On the provider chain, evidence expires after 80 days so only equivocation in the last 80 days can be punished.

In the provider chain, both downtime and double sign slashing happen automatically, there is no equivocation governance proposal. So how does this upgrade have anything to do with provider chain slashings if this is only related to removing double sign equivocation proposals for consumer chains?

@btruax

On consumer chains, evidence does not expire so any past equivocation can be punished. However, after key assignment, the previous key is only held in state for three weeks (21 days), so if a new key is assigned, any equivocation that occurred more than 21 days ago cannot be punished.**

This doesn’t seem correct. Let’s assume today 11th November Pupmos and Citizen Cosmos do a key reassignment, until the 2th December the previous key would still be held in state (21 days). If this v14 upgrade is implemented before the 2th December, then they would be tombstoned? You said ‘if a new key is assigned, any equivocation that occured more than 21 days ago cannot be punished’ but the double signing on Neutron of Citizen Cosmos and Pupmos happened more than 21 days ago, and in this example even if they do key reassigment today if v14 is implemented before 21 days from today they would be tombstoned? Could you clarfiy this @jtremback?

[lexa]

@Cosmic_Validator

So far there are only two consumer chains, Neutron and Stride. @lexa @jtremback you don’t have the information about all testnet names for Neutron and Stride before joining as consumer chains? With this information there can be certainty whether some old Neutron or Stride testnets had the same chain-id and whether there is risk about this or not, and if there was indeed some old testnet with same chain-id then all validators should be informed/reminded about this immediately.

Neutron and Stride have not had replicated security testnets with the same chain-id as mainnet. But while Hypha administers the Cosmos Hub testnet system, this concern applies to any cosmos sdk testnet. I can’t confidently say there has never been a testnet with the chain-id neutron-1 or stride-1 so we give the warning just in case.

However, long before this feature was developed, we did host Game of Chains and had a Noble testnet called noble-1, which is the same mainnet ID that the Noble chain uses. We expect Noble to put up a proposal to join the AEZ soon, so any participants from Game of Chains need to be absolutely sure they’re not reusing a key from that chain. The easiest way to do this is simply to use a fresh set of keys for mainnet noble-1.

I want to be super clear that no one did anything wrong in the naming conventions for this testnet and mainnet chain. These decisions were all made well before cryptographic equivocation was developed. We’ll do a fresh push of comms on this specific issue if Noble joins the AEZ.

Does this only affect @serejandmyself and @pupmos? Because so far there was only one consumer chain double sign equivocation proposal on the Cosmos Hub.

These are the main concerns, yes. We’ll be talking with these validators directly. However, it’s unlikely but technically possible that there was unnoticed double-signing so again, we give the warning just in case

You said ‘if a new key is assigned, any equivocation that occured more than 21 days ago cannot be punished’ but the double signing on Neutron of Citizen Cosmos and Pupmos happened more than 21 days ago, and in this example even if they do key reassigment today if v14 is implemented before 21 days from today they would be tombstoned? Could you clarfiy this @jtremback?

While the old key is in use or held in state, any equivocation that occurred using that key can be punished. When a new key is assigned, the old key is held in state for 21 days, during which time any equivocation using the old key can still be punished. Once 21 days pass and the old key is no longer held in state, equivocation using that key cannot be punished.

I do think the way we wrote that initial paragraph is incorrect @jtremback can you confirm and then @btruax, can we replace our comms paragraph with the thing I just wrote above ^

[serejandmyself]

@lexa would be good to hear something. for now it just seems that this might be the end for our validator node

PS. (Possibly ((since there is no clear understanding)) feels a bit like being punished for a crime, for which the law was passed after the crime was committed - i thought this didnt hold heh)

[lexa]

Hey!

You’re on my to-do list for getting in touch with on Monday

No, it won’t be the end of your validator node. There is a very clear understanding, which is that you should assign a new key for your Neutron node ASAP if you haven’t done it already. Once you assign a new key, it begins the 21 day period in which your old key is held in state and equivocation can still be punished.

The v14 upgrade is not expected to take place within 21 days, so by the time this feature is online, your old key will be cleared and there should be no risk of punishment for past equivocation. We are absolutely keeping your situation in mind as we schedule this upgrade.

Hope this clears things up, and please reach out to me on Discord or Telegram (@lexamichaelides for both) if you have questions or concerns.

Edit to add: Jehan has already spoken with Pupmos about this, so I consider both validators to have been informed now

[mrlp4]

@lexa @btruax i guess this should not go live on chain until 100% confirmed that past case (voted in prop 818) is completely worked out by affected validators, e.g. until confirmed that pumpos and citizencosmos have changed their affected keys and cool-down period of 21 days is complete before the upgrade block. Otherwise it will repeat one of the worst case on humanity practices of punishing the actions that took place before those actions was recognized as prohibited by whatever law

[lexa]

The forum post is the first step in a process that generally takes three weeks at a minimum (1 week on the forum, 2 weeks voting, + whatever buffer time is involved for feedback and reaching a Wednesday upgrade height).

Pupmos is confirmed to have changed keys already, and I’m waiting to hear back from Citizen Cosmos now. But it is a weekend, and there’s plenty of time before this goes on chain and we have to pick an upgrade height.

[serejandmyself]

No stress Lexa. Its just sometimes things move fast. And in theory, as @Cosmic_Validator pointed out, it could happen. That kinda shocked me. @mrlp4 already described perfect my feelings about it. Maybe its the way it was worded…

PS. @lexa i can confirm too now

[Cosmic_Validator]

@btruax

If you’re using old keys on consumer chains, you can eliminate all risk by doing a consumer key reassignment. Here are the docs on how to do a reassignment. You should do this ASAP.

@lexa

No, it won’t be the end of your validator node. There is a very clear understanding, which is that you should assign a new key for your Neutron node ASAP if you haven’t done it already. Once you assign a new key, it begins the 21 day period in which your old key is held in state and equivocation can still be punished.

@lexa

Pupmos is confirmed to have changed keys already, and I’m waiting to hear back from Citizen Cosmos now. But it is a weekend, and there’s plenty of time before this goes on chain and we have to pick an upgrade height.

Since Pupmos changed keys already, it seems the wording then is incorrect here in the documentation?:

Some validators seemed to be concerned that key reassignment was not possible now for Stride or Neutron.

[freak12techno]

I have a question not related to the upgrade, but related to equivocations and double signs on consumer chains. What concerns me is that on Neutron we are currently having the quite inconsistent situation: there are 2 validators who are marked as tombstoned (as they double-signed on Neutron chain), yet they are in fact not (as they weren’t tombstoned on Cosmos Hub chain and as Neutron takes the validators set out of Cosmos Hub, and the governance voted against slashing them). That might cause inconvenience, for example, on explorers when listing validators, some explorers might not display a validator in the active validators list as it’s tombstoned, while actually it is active and signs and produces blocks.

Are there any plans to resolve this inconsistency, for example, by having a Neutron chain upgrade that reverts these two tombstones (as the governance voted against slashing these validators)?

Not sure who to address this question to though, @lexa maybe you can answer?

[serejandmyself]

It really isn’t about that for me for example. It is really about much larger, and way more important things than running A node, or being slashed. Tried to express my thoughts here

Also, C.C. has reassigned keys too since yesterday

[serejandmyself]

There are other interesting things to go here. For example on Stride, it wont allow you to delegate LSM to Citizen Cosmos, as it states that the validator was “slashed” recently (there is only one occasion where the double sign occurred for us). There is a lot more related to that, but this is one of those weird occurrences that dont match the whole puzzle together.

[freak12techno]

Agree, also quite a valid example. I am pretty sure there are more cases like that I do not know about, that’s why I raised this concern here, this inconsistency makes it difficult to work with.

[lexa]

@serejandmyself – can you TG or Discord me (@lexamichaelides) so we can investigate some of this stuff and talk about the Neutron keys? I’m talking to the dev team and want to make sure we’re all 100% on the same page.

[Flo]

If 5% commission isnt enforced we shouldnt set this parameter to 5% at all. It would be a disadvantage in competetion for every validator that is joining after that change and secure a low commission monopole for earlier validators.

Actually I think this is a design flaw that should not be part of the v15 upgrade.

[mpoke]

@Flo We do plan to write for the v15 upgrade the migration code that will enforce the 5% minimum commission.

[Flo]

Okay thank you for the clarification.

But in case we increase the min commission in the future via governance we would have to enforce it again during a upgrade?