Thanks for continuing to manage this @Gavin. I feel now’s a bit pre-mature to review this in-depth, because many of the details are TBD at the moment.
Here are a few high-level thoughts in the meantime -
1 - The correct s/w version seems to be in flux. The current recommended production version is v0.34.9. However it’s not expected to last long, as another bug has been discovered and a fix is under development.
2 - We should fully test the upgrade process on a testnet, before a vote takes place to upgrade mainnet. This way we can have more confidence in the s/w and our vote. I also suggest following a process like @b-harvest organized.
3 - I feel the upgrade should only be considered successful after maybe 1000 blocks. 1 block isn’t enough. For example, the network hung after a block for many validators during the independent testnet @b-harvest organized. Also, in the past, test nets crashed on a regular basis within approximately the first 30 mins of the network coming online.