Hypha Quarterly Report: 2025-Q2

Conversation Proposal updates
1

Context

Proposal 985 ratified Hypha as the Hub’s testnet and mainnet operations team for 2025, with a budget of 650k from the community pool. This continued Hypha’s role as a community-funded team, carrying on our work from 2024. Our mandate this year is to steward the Hub’s testnet, train validators, coordinate mainnet upgrades, provide ICS support for new and existing consumer chains, and work closely with the Interchain Labs team.

This report summarizes our work throughout 2025-Q2 and plans for the coming quarter, per our commitment to transparency reporting on work performed with community-disbursed funds.

You can give us feedback on our work via our form here.

:star: Highlights

Reduced governance voting period to reduce need for emergency coordination

  • The Hub has had six emergency upgrades in the last six months, many of which have required a chain halt faster than the existing governance cadence can accommodate
  • Coordinated chain halts without using governance are demanding and risky for validators and the Hub as a whole - any time a chain halts, there is the risk that something will go wrong and it will not start again
  • There are many potential solutions to the problem of slow and risky emergency upgrades, but most of them require some level of development on gaia, the gov module, or the cosmos sdk, which delays solving the problem
  • As stewards of good mainnet operations, Hypha put forward a proposal to decrease the standard voting period from 14 days to 7 days and the expedited voting period from 7 days to 3 days
  • This proposal was passed on a 6 month trial basis, with Hypha committing to monitoring the governance reaction and put up a reversal proposal if needed

Performance testing for the cosmos sdk on the provider testnet

  • Spun up a fresh testnet with 35 equitably staked validators to load test and find performance improvement opportunities in Cosmos SDK
  • Identified an improvement opportunity in clearing and reusing caches for transactions - our preliminary tests suggest around a 20% improvement in memory allocation from this
  • We did a second load test with our testnet validators and recorded an improvement of 35% in terms of successful transactions under load using identical settings to the first test, but there is still room for improvement
  • Implemented a fix in the Cosmos SDK codebase, which is under review
  • We’ll keep working on stuff like this even though it’s technically outside the scope of our Hub contract - performance improvements in the stack benefit everyone’s operations

Forking tool update to accommodate migrations related to the x/lsm module deprecation

  • The migration logic required to move data from x/lsm to x/liquid were not compatible with the forking tool, blocking stateful upgrade tests. This is because the forking tool was set up to clear the entire validator set, which resulted in panics on startup when dealing with forks that had validators with x/lsm state.
  • An option to replace a single validator instead of the whole validator set by specifying the operator and consensus address was introduced to the forking tool.
  • The updated forking tool was successfully used to test the v24 and v25 upgrades with mainnet state. It will continue to be used to test future upgrades moving forward.

Upgrade and halt recovery drills

  • Several Testnet Tuesdays were dedicated to running drills focused on avoiding a halt during an upgrade, as well as recovering from a halt.
  • The first drill involved two back-to-back upgrades on an ephemeral network: a governance-gated upgrade followed by a non-governance gated one, and submitting a consensus-breaking transaction afterwards. Validators earned rewards for having upgraded before the consensus-breaking transaction went on-chain.
  • The second drill involved orchestrating a chain halt due to a consensus-breaking transaction on an ephemeral network.
    • The whole validator set started with Gaia v22.0.0.
    • Validators representing half of the voting power were asked to upgrade to Gaia v22.1.0 at a specific halt height, and the remaining validators were asked to stay with v22.0.0.
    • After the consensus-breaking transaction was submitted, the validators that were still running v22.0.0 were instructed to rollback one block, upgrade, and rejoin consensus.
  • The third drill followed the same process as the second one, with the roles swapped among validators so everyone would have a chance to practice the rollback command in a live public network.
  • These drills helped many validators tackle the v23.1.1 upgrade halt in mainnet with confidence, having just gone through a very similar scenario in a controlled, “low-stakes” setting less than a month earlier.

:boxing_glove: Challenges

Chain halt (v23.1.1)

  • On May 9, 2025, a coordinated emergency upgrade on the Hub led to a chain halt for about four hours while Hypha, the Hub validators, and the Hub team at ICL worked together to bring the chain back
  • Many Hub validators were prepared to use the rollback command effectively from educational events on the testnet
  • A retrospective on the security patch that triggered this upgrade is available from the IBC team here
  • A retrospective on the upgrade process on the Hub is available from Hypha and ICL here
  • From Hypha’s action items:
    • We’ve successfully reduced the governance voting period params
    • We’ve investigated using a multiplexer feature (like Celestia does) for halt-less upgrades and put together a proposal for how this could be integrated into the sdk or gaia. This work is on-hold pending the dev team’s priorities.
  • Shared action items are still in progress, pending other priorities.

Pause of Testnet Incentive Program funding

  • AADAO’s funding of the Testnet Incentive Program has ended as of June 30, 2025.
  • The testnet program will continue with a focus on education and validator-oriented events that will not be incentivized.
  • We expect to see some validators leave the program without incentives being paid, which means that the validator set will shrink and we will have access to fewer validators to replicate the variance in validator set up that we see on mainnet.
  • Continued funding from ICL is in-progress and will mean revamping the incentive structure to align with ICL’s priorities.

Operational Report

Mainnet coordination

Testnet events

  • Jun 17: Provider upgrades to gaia v25.0.0.0-rc0
  • Jun 10: Gaia v24 changelog review
  • Jun 3: Performance load testing with cosmos sdk fix
  • May 27: Provider upgrades to gaia v24.0.0-rc1
  • May 20: Provider upgrades to gaia v24.0.0-rc0
  • May 23: Provider upgrades to gaia v23.3.0
  • Apr 29: Provider upgrades to gaia v23.1.1
  • Apr 22: Performance load testing baseline
  • Apr 15: Observability tooling demo and setup
  • Apr 8: Information gathering for build environment
  • Apr 1: Upgrade drill using rollback command
  • Apr 1: Provider upgrades to gaia v23.0.1

Financial updates

Our cashflow report is very simple this year as we are billing a consistent amount each month and not processing any returns.

For transparency, we hadn’t linked our withdrawal tx for Mar 2025 as it hadn’t happened yet at the time we posted our last update. For Mar 2025, we billed 54,166.67 USDC withdrawn on May 15 (tx here).

  • Apr 2025: Billed 54,166.67 USDC, withdrawn on May 15 (tx here).
  • May 2025: Billed 54,166.67 USDC, withdrawn on Jun 5 (tx here).
  • Jun 2025: Billed 54,166.67 USDC, no withdrawal yet.
  • Remaining USDC: 376,839.723642

During this quarter, we were also able to liquidate our remaining ATOM in two different deals through our CEX, BitBuy, to fund the remainder of our year in USDC.

  • Sold 24,956.51349683 ATOM at a price of 5.00871 USDC to get 125,000 USDC
  • Sold 61,581.84000000 ATOM at a rate of 5.24117 USDC to get 322,761.09047861 USDC

Plans for Next Quarter

Generally speaking, Hypha’s plans for next quarter will be the same each time as the maintenance and production operations team:

  1. Test upcoming Gaia releases
  2. Schedule and coordinate all mainnet upgrades (gov, non-gov, and rolling)
  3. Manage the provider testnet based on the dev team’s needs
  4. Offer educational and performance-focused events on the testnet for Hub validators

In addition to our usual operations, we are specifically planning on preparing and launching the third iteration of the Testnet Incentives Program based on product needs of the Hub, developing our mainnet smoke test set for reviewing basic transaction functionality on each upgrade, and helping integrators test their product on the testnet.

Feedback and Responses

We have not received any direct feedback yet! You can offer us feedback via our form here.

As a reminder, our mandate on the Cosmos Hub is to:

  1. Steward the Hub’s testnet as a platform for validator education, final runway for Hub upgrades, and stable development platform. The provider testnet’s purpose is to serve the Hub’s direct needs.
  2. Provide mainnet and Interchain Security coordination support for validators and consumer chains.
  3. Provide development support for the Hub team at Interchain Labs.
4 Likes
2

Thanks.
About FORGE,Are there any improvements planned? We notice that Forge has disappeared from the Cosmos website and been replaced by “solution for institutions”? A few words on that?
Are new chains in the process of joining ICS (Intento for example)?
The development of Mesh Security has been granted and retained. Are you on the case?

1 Like
3

Hello! Thanks for sharing!

1 Like
4

Thanks for the report Lexa

1 Like
5

Forge is/was a core part of Interchain Security as the product vision for the Hub and was the dev responsibility of the last product team.

My assumption (not official, not speaking on behalf of either Hypha or ICL) is simply that Forge was Informal’s project to support the product vision they put forward, and further developing and promoting that product vision isn’t on the roadmap going forward.

Hypha specifically is not developing/maintaining Forge and we are not working with any new chains looking to onboard through ICS, though we are still supporting Elys and offer support to any of our existing consumer chains. Not sure about Mesh Security - that wasn’t part of our mandate as a maintenance team.