Hypha Quarterly Report: 2025-Q1

Conversation Proposal updates
1

Context

Proposal 985 ratified Hypha as the Hub’s testnet and mainnet operations team for 2025, with a budget of $650k from the community pool. Our mandate this year is to steward the Hub’s testnet, train validators, coordinate mainnet upgrades, provide ICS support for new and existing consumer chains, and work closely with the Interchain Labs team.

This report summarizes our work throughout 2025-Q1 and plans for the coming quarter, per our commitment to transparency reporting on work performed with community-disbursed funds.

You can give us feedback on our work via our form here.

:star: Highlights

Provider testnet supports the IBC-Go team to bring IBC Eureka to the interchain

  • The provider testnets’ prime directive is to serve Gaia’s needs, which does not mean testing for stack bugs or pre-release versions (e.g. cosmos sdk, ibc, cosmwasm).
  • IBC Eureka is a core part of ICL’s plan for the Hub going forward, but the integration was blocked by having no live testnet exposure for IBC v10.
  • Hypha pushed the provider testnet through many fast pre-release upgrades to give the IBC-Go and Hub team confidence in IBC v10.
  • Hypha coordinated a mini testnet with some technically skilled validators to perform an upgrade test leading up to v23 to build confidence in the upgrade path even without a full testnet upgrade.
  • v23.0.1 upgrade was ultimately successful on the Hub and Eureka is now live!

Website launch of testnets.hypha.coop

  • Our team has always been very heads-down and focused on getting things done, which means that some of our most valuable resources remain hidden and hard-to-find.
  • testnets.hypha.coop houses the points tally for our Testnet Incentives Program events so that validators can track their progress.
  • Also integrates our exceptions form, registration form, and feedback form for the Hypha team.
  • Future plans include write-ups of our educational events so the content is more accessible.

Upgrade drills on the provider testnet

  • So many emergency upgrades in Q1 has prompted ICL and Hypha to think about ways to smooth out the process
  • In the meantime, we are working with the testnet validator set to drill different upgrade scenarios
    • Non-governance coordinated upgrade with cosmovisor (this is what we usually do on mainnet)
    • Halt and recovery (practicing how to come back from a halted chain with < ⅔ of the set upgraded)

:boxing_glove: Challenges

Failed rolling upgrade (v22.0.1)

  • First emergency upgrade with the ICL team doing the integration and Hypha running coordination.
  • Mis-categorized as a non-breaking upgrade and instructed validators to upgrade on a rolling basis.
  • Quickly rolled back the upgrade (well below ⅓ VP) and delayed the upgrade until a coordinated block height could be chosen the following week.
  • Retrospective hosted by Hypha and published by ICL here.
  • Many lessons learned:
    • Careful review of patch notes for new dependencies
    • Live experience with non-gov coordinated upgrades - they’re very difficult to coordinate
    • No upgrades on Friday :upside_down_face:

Last minute binary swap to address build issues in v23.0.0 (v23.01)

  • Last minute reports from validators about build errors with the v23.0.0 binary < 24 hours before upgrade height.
  • We investigated and identified an issue that needed to be patched prior to upgrade.
  • Late night work from both Hypha and ICL patching the binary, testing, and releasing information about the binary swap to validators.
  • Hypha has hosted and posted a postmortem for this issue here.

Many emergency upgrades

  • Natural result of a new team digging into the stack - lots of bugs identified and fixed!
  • An emergency upgrade nearly every two weeks is hard on the schedule and on validators.
  • Upstream coordination will reduce this cadence going forward - thank you, ICL team :folded_hands:
  • Lessons learned on the Hub team:
    • Need a way for validators to signal a successful upgrade for coordinated and rolling upgrades that don’t use an upgrade handler. This has been acknowledged by the Hub team!
    • In the meantime, need to build the muscle of validators manually confirming via Discord or filling out a form. We continue to remind validators and explain why this is necessary.
    • Need for a clear “upgrade captain” role of someone who is running all coordination and communication - can’t always rely on the same person, so the process needs to be clearly defined and shared amongst the team. This process has now been documented for all major upgrade types (gov, non-gov, rolling, halt and recover).

CryptoDungeon offboarding

  • Dungeon got offboarded in January due to a timed out CCV channel.
  • Hypha replicated and mapped out the process for rejoining the Hub if a chain offboards itself accidentally (this is not something already accounted for in code or documentation so we were making this up from scratch).
  • The process involves patching the consumer chain via a manual binary swap so that the chain is able to execute a governance-gated software upgrade, as the consumer chain is not currently capable of doing this.
  • The Dungeon team has chosen to fork and exit the Hub’s security instead - we wish them all the best!

:clipboard: Operational Report

Mainnet coordination

  • Upgrades
    • V22.0.0 (regular governance)
    • V22.1.0 (emergency coordinated)
    • V22.2.0 (emergency governance)
    • V22.3.0 (emergency coordinated)
    • V22.3.1 (emergency rolling)
    • V23.0.0 (regular governance)
  • Security events impacting the Hub

Testnet events

Financial updates

Our cashflow report is very simple this year as we are billing a consistent amount each month and not processing any returns.

  • Jan 2025: Billed 54,166.67 USDC, withdrawn on Feb 20 (tx here).
  • Feb 2025: Billed 54,166.67 USDC, withdrawn on Mar 18 (tx here).
  • Mar 2025: Billed 54,166.67 USDC, no withdrawal yet.
  • Remaining USDC: 91,666.66
  • Remaining ATOM: 86,538.46

Since the time of our proposal, ATOM has dropped below the $5.2 threshold that would allow us to liquidate and receive the full budget outlined in our proposal.

Rather than liquidate into USDC at a loss to both the community and us, we will accept the entirety of the 86,538.46 ATOM as our budget (currently valued at ~$415k rather than the $450k in our proposal). This will allow us to potentially stake and contribute to the Hub’s security, receive staking rewards that will bridge the gap between our budget and the current value of ATOM, and make strategic decisions for how and when to liquidate so that our budget is met and the Hub continues to receive the full scope of our work outlined in the proposal.

:spiral_calendar: Plans for Next Quarter

Generally speaking, Hypha’s plans for next quarter will be the same each time as the maintenance and production operations team:

  1. Test upcoming Gaia releases
  2. Schedule and coordinate all mainnet upgrades (gov, non-gov, and rolling)
  3. Manage the provider testnet based on the dev team’s needs
  4. Steward the Testnet Incentives Program

In addition to our usual operations, we are specifically planning on the following work:

  1. Improved redundancy to communications and upgrade procedures
  2. Performance testing and improvement on Gaia codebase
    1. We have developed a load testing and observability stack that can help us diagnose performance bottlenecks in cosmos SDK codebases
    2. We intend to leverage our existing group of validators to load test at realistic scales
    3. There are several potential areas for improvement in transaction processing that could result in performance gains. We don’t know yet what kind of results to expect, and results will be conditioned by on and off-chain configuration parameters and the exact shape of the load
  3. Alternate funding plans for the Testnet Incentives Program, as AADAO’s grant will end

:horizontal_traffic_light: Feedback and Responses

We have not received any direct feedback yet! You can offer us feedback via our form here.

As a reminder, our mandate on the Cosmos Hub is to:

  1. Steward the Hub’s testnet as a platform for validator education, final runway for Hub upgrades, and stable development platform. The provider testnet’s purpose is to serve the Hub’s direct needs.
  2. Provide mainnet and Interchain Security coordination support for validators and consumer chains.
  3. Provide development support for the Hub team at Interchain Labs.

Informal feedback we have noticed about our work:

  1. Our Testnet Incentives Program announcements in Discord do not always clearly identify the time range in which points can be earned. We have adjusted our process to give timestamps on when points are available rather than when we will be online and running a synchronous event.
  2. Hypha’s whale validators on the testnet make it unrealistic for testing mainnet’s behaviour under certain conditions. This is a tradeoff between realism and agility - our whale validators allow us to make rapid changes to the testnet based on the Hub dev team’s needs (e.g., putting provider through three release candidates in less than a week) without needing to coordinate 60 validators to get online, vote, and apply an upgrade on an unreasonably fast timeline. Rapidly accommodating the dev team’s needs is currently the priority for the testnet.
  3. Testnet upgrades should be announced well in advance so validators have time to prepare. We agree! Currently, Hypha’s validators are able to execute an upgrade without any other validators needing to be around. Announcements for last minute upgrades serve to inform validators of what is happening so they can catch up when online again, but they are not expected to respond to last minute upgrades on the testnet, nor do we incentivize them via TIP. We have no intention of demanding that validators remain on-call for the testnet.
3 Likes