February 14, 2024 
v15 is coming to the Hub! This is going to be a big one because of state migrations (executing on decisions made in proposals 826 and 860) and moving to sdk 0.47.
On the testnets, we upgraded theta-testnet-001 on Tuesday and did provider today to ease the load on validators operating on both networks. We had 41 validators participating on provider.
- Theta runtime: 55 minutes
- Provider runtime: 41 minutes
Lessons learned
Major feedback from operators who have now gone through this upgrade process at least once (some went through it twice):
- Don’t panic at the long wait and lack of updates in the logs! Logs are not very verbose and it will look like nothing is happening, but just be patient. Hypha and Informal are working with the Comet team to see about getting more informative logs during upgrades in the future.
- Do not restart your node mid-upgrade unless you’re seeing OOM errors. Having learned from our big Neutron upgrade, we know that restarting mid-upgrade can make the situation worse and lead to a corrupted database.
- Set min gas price to
0.005uatomor your node will fail on restart. Failing on restart just means that you’ll have to set the param at that point, but doing it ahead of time will be smoother
Our testnet work today fed directly into the mainnet upgrade recommendations, such as ensuring that some validators will be taking snapshots to help validators who aren’t able to meet the hardware requirements required for mainnet state migrations.