Testnet Wednesday Reports

Stride Launch Rehearsal (July 5, 2023)

This Wednesday, Hypha hosted Stride’s second launch rehearsal on the Replicated Security Persistent Testnet immediately after a very smooth Neutron mainnet upgrade :tada:

The testnet program is stewarded by Hypha Worker Co-op and provides a testing environment for pre-launch consumer chains on the Replicated Security Persistent Testnet. This work supports the success of the Cosmos Hub mainnet by catching issues before they go live and letting dev teams and infrastructure providers alike practice with the new technology introduced by RS. Stride will be the second Hub consumer chain to go through launch rehearsals on the RS testnet and the first sovereign-to-consumer transition.

With Stride going live on the Cosmos Hub in two weeks, the stakes were high for this rehearsal. In the last month, Hypha and the Stride team have worked closely to iterate on feedback from our last rehearsal on June 7.

Naming convention clarity

For chains that start off as consumer chains (such as Neutron), a genesis file is provided ahead of time and then modified after spawn time to include the CCV state. This modification can only happen after spawn time, as the CCV state is generated by the provider chain at spawn time, which is set in the consumer-addition proposal.

Chains that transition from sovereign to consumer (such as Stride) already have a genesis file in-use and need to have the CCV state added to the binary in a new file. After some confusion in our first rehearsal, Stride and Hypha have decided to adopt the terminology:

  • genesis file is the file used to originally start the sovereign chain. It does not contain the CCV state.
  • ccv file is the file that is generated after spawn time and which contains only the CCV state. It is added to the binary and does not replace the existing genesis file.

Spawn and upgrade time buffer

In the time between spawn on the Hub and upgrade on the sovereign Stride chain, validators need to generate the CCV state and append it to the pre-existing genesis file associated with Stride, then restart their nodes with the post-transition binary. In our first rehearsal, we only had 2 minutes to do this!

This time, we had about 20 minutes between spawn time and upgrade time, which made it much easier for validators to bring their infrastructure online without scrambling or feeling rushed. Feedback

Having this buffer time (or more) is strongly recommended for Stride’s mainnet launch.

Outreach and communication

In the weeks leading up to this rehearsal, we publicized many announcements via email, Discord, Twitter, and Telegram to bring new validators into the rehearsal.

We currently have 48 active validators on the Replicated Security testnet, 16 of whom are also validators on Cosmos Hub and Stride mainnet. Having these double mainnet validators participate in rehearsal will make sure that some of the most invested stakeholders will be prepared and experienced when it comes time to do it on mainnet!

Because this is the first time a chain will transition from sovereign to consumer, the process is still new to everyone and lots of effort went into documenting the steps to improve validator awareness and smooth out the transition.

  • Detailed post outlining steps in the transition
  • Diagram of the changeover process created by Stride
  • Step-by-step sequence table shared in Discord announcements

Consumer key assignment

We also continued to provide clarity on how the consumer key assignment feature works!

Consumer key assignment transactions must happen before spawn time. For Stride on mainnet, that means the transaction has to occur before July 19, 2023.

If the consumer key is not assigned, validators must reuse the provider key until a relayer is established and a new consumer key assignment transaction can be sent via the relayer.

Get involved with the Cosmos Hub testnet program!

If you have questions, comments, or feedback, we want to hear from you! Connect with us on Discord in the #replicated-security-testnet channel or DM me on Telegram (@lexaMichaelides) or Discord (lexamichaelides).

Are you a validator or node operator who is not yet part of the Replicated Security testnet? Join us!

3 Likes

Conversation of here proposals need voting before approved.

We’re going to use this thread as a running log of Testnet Wednesday events!

July 12, 2023

We launched Duality’s second chain on the Replicated Security testnet on July 12. duality-testnet-1 will remain online as Duality’s persistent test chain. The launch event took about one hour from proposal passage until interchain secured.

Shortly after spawn time, we ran into a JSON unmarshalling issue that caused the Duality binary to panic. After a bit of digging, we discovered that the app_state.group value had been incorrectly defined as an array of objects, rather than one object. With the aid of a few helpful validators, we fixed the bug, published an updated final genesis file, reset the node state, and started seeing blocks.

Today’s launch also reminded us to always run genesis files through jq before generating checksums. Without standardizing the format of genesis files with jq, whitespaces and differently-ordered keys will lead to different checksums, even though the data is the same. This is especially important considering that some validators build their own final genesis files after spawn time.

We unfortunately did not manage to have enough time to upgrade pion-1 today, so we will be upgrading the Neutron persistent chain at a later date.

Thank you to all participating validators, and to the Duality team for their patience with having this rehearsal rescheduled a bunch of times. Keep an eye out for Duality’s on-chain proposal coming soon!

Validator shout outs :raised_hands:

  • KrEwEdk0 from CitadelOne for their help debugging the consumer chain genesis file
  • 20/20 from VirtualHive and KrEwEdk0 for sharing their address books, getting many other validators online quickly
  • James from LavenderFive for flagging the bug with the final genesis file early
2 Likes

July 26, 2023

We had THREE testnet upgrades in one day! This is the highest number of operations performed in one day so far.

  • Upgrade theta-testnet-001 to v11
  • Upgrade provider to v11
  • Upgrade pion-1 to v1.0.4

The Gaia upgrades on the Theta testnet and the provider chain went smoothly. Both were major version upgrades, and these were coordinated upgrades via governance proposal. In both cases, the chain came back online within a few minutes after hitting the upgrade height. pion-1’s block height also drifted forward a bit, enabling us to start and conclude all three events in under 90 minutes. Snappy!

Coincidentally, we observed an issue with Hypha’s validators around the same time as the first scheduled upgrade. On the Replicated Security testnet, the nodes that were running Stride ran out of memory. This led to Hypha’s Stride validators going offline for a bit, which led to a temporary loss of consensus on the provider chain. Thanks to alerting from some proactive validators, we were able to provision more memory and recover in time for the Gaia upgrade.

Validator shout outs :raised_hands:

  • Bosco from Silk Nodes for posting step-by-step instructions for the Gaia upgrade
  • Alan from Nodestake for promptly uploading post-upgrade snapshots for both the Theta testnet and the Replicated Security testnet
2 Likes

Wednesday August 23

The last few weeks have been pretty quiet on the testnets! Here’s a quick roundup.

August 9, 2023:

  • No coordinated Testnet event, but all validators were asked to upgrade their Gaia version from the v11 release candidate to the official v11 release.

August 16, 2023:

  • No testnet event, but we upgraded mainnet to v11.

August 23, 2023:

  • Upgraded release testnet from v11 to v12-rc
    • Upgrade height at 13:41 UTC; blocks at 13:47 UTC
  • Upgraded provider chain from v11 to v12-rc
    • Upgrade height at 14:20 UTC; blocks at 14:25 UTC

Very boring and smooth upgrades all around! Thanks everyone for your help in making this another successful Testnet Wednesday :ribbon:

2 Likes