Use governance for emergency upgrades

Top-level post is intended to collect relevant info! Here are my personal opinions:

Validator attention

The Hub has a very responsive validator set that can be counted on in emergencies. I wouldn’t want to abuse that responsiveness by introducing a lot more rapid governance processes, however I prefer (and I think validators would prefer) rapid governance → on-chain data for emergency upgrades over rapid off-chain coordination via email and Discord.

Below, I highlight the risk of exposing vulnerabilities and wonder about setting up an empty upgrade proposal (pointing at an empty repo) and then adding the binary 24h in advance of the upgrade height. If this is possible (and acceptable, from a software dev and social norm perspective) then I think the governance doesn’t actually have to be ‘rapid’ after all. We could probably use the same voting period as a normal software upgrade (whether that’s 1 or 2 weeks).

Risk of exposing vulnerabilities

Imo, the biggest risk in using governance for emergency upgrades is when we are fixing a major security vulnerability.

In this case, we need to publish the fix and have it go through governance before doing the upgrade. The worst case Ontario is that this fix is published (exposing the vulnerability) but the upgrade doesn’t actually pass and the chain doesn’t actually upgrade. Terrible!

@mpoke – would it be possible to put an upgrade on chain and point it at a repo where the fix will be published 24h in advance of upgrade height, the way we currently do for coordinated emergency upgrades? Like we set up the upgrade proposal to happen but don’t actually reveal the binary until we know the proposal is going to pass and it’s safe to reveal the fix.

Shift to using full semantic version names

I’d also welcome Marius’ thoughts on shifting from naming versions using only the major version to using the full semantic name. I believe Neutron does this (example) and thus uses governance for all upgrades, but it also introduces the risk of human error in failing to keep the release and upgrade name in sync (example from Neutron).

1 Like