Skip has announced their acquisition by the ICF and the formation of a new entity: Interchain Inc. Under Interchain Inc., all aspects of the core stack will be brought together for in-house work, including the technical development and stewardship of the Cosmos Hub. This is an exciting time for the Hub – strengthening its position as the gateway to the interchain under new leadership.
Our work remains focused on supporting the Hub with our well-established testnet program and coordination experience.
We’ve waited to post our 2025 funding proposal until this news became public, so the community has access to the full context of how Hypha and the Hub team at Interchain Inc. will be able to work together.
Our plan is to put this prop on-chain at the start of the new year to allow time for feedback and so as not to interrupt the holidays. This means there will likely be a 2-3 week period in January 2025 where Hypha’s mandate and funding has ended. During this time, we intend to continue the work we’ve planned (e.g., ensuring the successful v22 upgrade) with the outgoing Hub team at Informal and the incoming team at Interchain Inc. We will also continue coordinating the Testnet Incentives Program and hope to continue that work under a new mandate for 2025.
Changelog
- 2024-Dec-16: Posted initial draft, added in-text navigation links
- 2025-Jan-06: Adjusted ATOM and USDC estimates, added wallet address, linked Q4 wrap-up report, and moved into last call
- 2025-Jan-09: Adjusted ATOM estimate before posting on chain
Fund testnets and mainnet ops by Hypha in 2025
Navigation
- Hypha’s past work on the Hub
- Proposed work: testnets
- Proposed work: maintenance and ops
- Proposed work: dev support
- Budget
- Logistics of funding process
- Accountability
Summary
The Hub testnets team at Hypha Worker Co-operative is seeking funding to continue their mandate of stewarding the Hub’s testnet, supporting mainnet operations, and development support and onboarding for the incoming Hub team at Interchain Inc.
This proposal is summarized by the following propositions:
- The approval of a $650,000 USD budget to finance Hypha’s:
- Stewardship of the Hub’s provider testnet as a dev platform and validator training program
- Mainnet coordination for upgrades and validators
- Coordination around bug fixes, security patches, and version updates for core technologies
- Mainnet ICS support for the Hub’s current and prospective consumer chains
- Validation testing for core stack integrations and new features
- Testing and development support for patches and version update releases
- Onboarding and transition support for Skip/Interchain Inc.
- The release of the following funds to a Hypha-controlled wallet:
- 200k USDC (shown in the spend proposal details as 200,000,000,000 IBC-wrapped uusdc)
- 86,538.46 ATOM (accounting for 69,230.77 ATOM at a spot price of $6.5 USD with a 25% buffer to cover price fluctuations) to be converted to $450k USDC via the process laid out in the full text of the proposal
- The approval of Hypha’s transparency and accountability commitment to the ATOM community:
- Quarterly reports made to the Cosmos Hub forum
- Quarterly withdrawal reports on community-disbursed funds
- An anonymous feedback form and commitment to publicly reviewing and incorporating relevant feedback
- Cooperation with the ATOM community if a signalling proposal is passed to return any unused funds
- Cooperation with teams chosen by the ATOM community to fulfil other scopes of work or to replace our services on the Hub
Governance votes and outcomes
The following items summarize the voting options and what it means for this proposal:
- YES: You agree to fund the Hub testnets team at Hypha Worker Co-operative with a $650k USD budget for continued testnet stewardship, mainnet coordination, and development support over the period of January 1, 2025 - December 31, 2025. A ‘YES’ outcome will immediately release 200k USDC and 86,538.46 ATOM to the specified wallet (
cosmos1ax7krw2ymq8upaxwce7df3xpalpp0jddm29fws). - NO: You do not agree to fund the Hub testnets team at Hypha Worker Co-operative based on the terms of this proposal.
- NO WITH VETO: A ‘NoWithVeto’ vote indicates a proposal either (1) is deemed to be spam, i.e., irrelevant to Cosmos Hub, (2) disproportionately infringes on minority interests, or (3) violates or encourages violation of the rules of engagement as currently set out by Cosmos Hub governance. If the number of ‘NoWithVeto’ votes is greater than a third of total votes, the proposal is rejected and the deposits are burned.
- ABSTAIN: You wish to contribute to the quorum but you formally decline to vote either for or against the proposal.
Hypha’s past work on the Hub
Hypha has been a core contributing team on the Cosmos Hub since 2022, focusing on the Hub’s testnet program.
Our work in 2024 was funded via Proposal #839 and comprised:
- Regular validator training on new Gaia releases and complex new features in Interchain Security
- Facilitating the Testnet Incentives Program to compensate validators for their hard work on the Hub’s testnets and address economic concerns for Interchain Security
- Mainnet coordination of validators and software upgrades (governance, coordinated, and emergency)
- Major upgrade testing and de-risking for all Gaia releases through local testnets, devnet, and public Hub testnets
- Interchain Security contributions to documentation, primarily for node operations with the release of Partial Set Security and permissionless ICS
- A rapid incentivized testnet (ISLE) to educate validators and incoming consumer chains on Partial Set Security before mainnet launch
Hypha has consistently been recognized as either meeting or exceeding expectations by our 2024 Oversight Committee in all categories. For detailed information, you can review our public quarterly reports:
- Hypha Oversight Meeting: Q1 Wrap-Up, Q2 Kickoff
- Hypha Oversight Meeting: Q2 Wrap-Up, Q3 Kickoff
- Hypha Oversight Meeting: Q3 Wrap-Up, Q4 Kickoff
- Hypha Oversight Report: Q4 Wrap-Up
This is a proposal to fund ongoing testnet work, mainnet coordination, and development support on the Cosmos Hub for 2025.
Proposed work in 2025
Stewarding the Hub’s persistent testnet (provider)
provider is a stable testnet that leads the Hub through upgrades and provides a training and rehearsal platform for node operators, consumer chains, and integrators working with the current version of Gaia.
In Hypha’s time stewarding the Hub’s testnets, we’ve had two different public networks: theta-testnet-001 and provider. In our efforts to streamline testing and improve the validator experience, theta has been deprecated as of December 1, 2024 so Hypha’s Hub testnet work will focus only on provider going forward.
The Hub’s testnet acts as:
- A training ground for validators and node operators
- A final runway to mainnet once other testing has passed
- A stable development platform for integrators and incoming projects
Training ground
As a training ground, the testnet serves validators interacting with new features.
Roughly 60 of the Hub’s active validators are present on the testnet and participate every week in events run by Hypha to educate and train them in aspects of ICS and Gaia. Consistent participation is currently incentivized via the Testnet Incentives Program, currently funded by AADAO.
In 2025, we intend to transform the Testnet Incentives Program to better align with validator needs. We will be using a points-based system to reward high performance and working with the ICF to incentivize testnet participation via the ICF Delegations Program on the Hub.
Some of the work involved in coordinating provider as a training ground:
- Creating and scheduling testing and demo events to expose features to live validators
- Training and teaching validators about Hub features
- Setting norms for good node operations on the Hub
- Troubleshooting and assisting validators with node operations
- Evaluating and disbursing incentive payments on a timely schedule
Final runway
As a final runway, the testnet serves all of the Hub’s stakeholders by de-risking upgrades and consumer chain launches.
Hypha has guided both testnets and the Hub through 15-20 upgrades and every consumer chain launch since the inception of ICS. Our expertise is the bridge between development and live users and contributes to the Hub’s record of easy and fast upgrades in 2024.
- As testnet validators go through an upgrade, Hypha collects feedback to prep announcements and guidance for mainnet upgrade
- Non-testnet validators gain confidence in a new release because it has already been exposed to many Hub validators and the upgrade path has gone smoothly
- The testnet serves as a final test that estimates the resources needed for migrations, and any issues with third party tools that might cause a bumpy mainnet upgrade
- All consumer chains have gained confidence in their code and appeal to validators by going through launch rehearsals guided by Hypha
Stable development platform
As a stable development platform, the testnet serves Hub integrators and developers (Gaia, ICS, CosmWasm contract devs, block explorers, wallets, etc).
The Hub is currently a platform for launching consumer chains and permissioned smart contracts. Development teams looking to launch on the Hub run rehearsals and test their compatibility with the Hub using the testnet. The testnet is also connected to many third-party integrators that rely on it to test Hub features: Mintscan, Big Dipper, Keplr, etc. These integrations also make it possible for less technical users to onboard to the Hub.
- Assisting consumer chains through launch rehearsals
- Providing a venue for integrators to test with live Gaia code
- ATOM holders learning (people testing proposals)
- Other testnets with IBC connections
Mainnet maintenance and operations
Validator coordination
As a trusted source of chain expertise and information, Hypha’s coordination work serves validators directly. It also serves all users of the Hub by maintaining smooth and reliable operations.
Hypha’s work with mainnet validators has been integral to the Hub’s smooth operations in 2024. Every upgrade has involved behind-the-scenes coordination with Hypha to ensure that the features are well understood and that node operators know what to expect when we upgrade.
This work includes:
- Scheduling upgrades to reduce conflicts with other chains and competing interchain commitments
- Communicating upgrade requirements (e.g., hardware, timing, any unusual operational needs)
- Answering questions and supporting validators with pre-upgrade operations using our expertise from performing the upgrade multiple times
- Being on standby during upgrades to troubleshoot, debug, and address any live issues
Incident response
Consultation and coordination work around security incidents serve the stability of the Hub and validators.
We have increased our involvement in security response work this year in collaboration with Informal and other Cosmos teams. Security issues are received and addressed by the development team with input from Hypha’s testing team. Once the issue has been patched, Hypha and the development team work together to produce and distribute information about the fix and necessary upgrades. Hypha’s incident response strengths are in analyzing the bigger picture and facilitating processes that are calm and coordinated amongst the validator set and with other chain teams as needed. We are also very good at writing timely retrospectives and incorporating our lessons learned going forward.
Hub development support
Validation testing (for features and integrations)
This testing checks that features behave as the user expects. It serves the users of the feature.
Hypha validates that the code behaves as users expect. This is important work for new features, but also necessary for the integration of changes from other modules included in Gaia (e.g., IBC, Cosmos SDK, feemarket, etc). The roadmap for the Hub in 2025 hasn’t yet been announced, but we’ll be ready to run our validation work on whatever the Hub team at Interchain Inc. puts forward.
Gaia devnet (for early releases)
This work is meant to give developers a live platform for their work. It serves the Gaia development team, and integrators can benefit from it as well.
In 2024, Hypha began running a devnet to test each major release of the Gaia codebase before a release candidate is tagged. This resource speeds up the development cycle by allowing developers to catch upgrade issues prior to cutting a release and having the full testnet validator set test it (and potentially find issues). Information about joining the devnet and interacting with it is available in the devnet repo.
Local testnet upgrades (for release candidates)
This testing is meant to ensure that mainnet upgrades go smoothly. It serves the validators.
Once Gaia versions are passing the initial suite of verification and validation tests, a release candidate is produced and Hypha runs several local upgrade tests to confirm that the upgrade path and basic “happy path” transactions are available immediately after upgrade.
- Major upgrades: A single-validator chain goes through a software upgrade via governance proposal using a matrix of all versions available at the time.
- Minor or patch upgrades: A single-validator goes through a manual upgrade (binary replacement) to check if the upgrade is immediately consensus- or state-breaking after upgrade.
- Fresh genesis (major/minor/patch): A new genesis file is initialized and the chain begins at height 1 using gaiad only, then cosmovisor with a manual binary upgrade, then cosmovisor with auto-download if the release is published.
- Stateful upgrade (major/minor/patch): A mainnet snapshot is modified to allow it to run with a single validator. A software upgrade proposal is submitted and passed and the upgrade is performed.
Operational details of funding
Budget
This proposal asks for a budget of $650k USD denominated in a mix of ATOM and USDC.
For context, in 2024, we were given a fee cap of $1.18 million USD. We expect to use approximately 85% of that amount (~$1mil). All unused funding is being returned to the community pool at the end of each quarter.
The 2025 budget is lower than our 2024 actuals, for a few reasons:
- We are not including any incentivized testnets such as Game of Chains or ISLE in this budget.
- We are not including any major feature testing at this time, as the exact features are not yet defined.
- Skip/Interchain Inc. may internalize some of the development support work that Hypha has previously done. In the early part of 2025, we plan on continuing this work to avoid an interruption of service, and then transitioning it fully to Skip/Interchain Inc.
| Service to be delivered | Cost |
|---|---|
| Hub testnet stewardship | $350,000 |
| Mainnet coordination | $200,000 |
| Development support | $100,000 |
| Total | $650,000 |
As a maintenance and operational service level agreement, the line items in this proposal have a set budget – no headcount estimates and no performance bonuses. The full budget will be invoiced and withdrawn by Hypha on a monthly basis.
Liquidation, withdrawal, and return process
This $650k budget is broken down into both ATOM and USDC from the community pool. The proposal type has been tested on the provider testnet to confirm that both denoms will be transferred correctly.
There is currently 200k USDC available in the community pool, meaning that the remaining budget to be covered in ATOM is $450k. At an ATOM spot price of $6.5 USD, this is 69,230.77 ATOM. To account for fluctuations in ATOM price over the course of the voting period, a 25% buffer is introduced on this ATOM portion, for a total of 1.25*69,230.77 = 86,538.46 ATOM. Unused ATOM will be immediately returned to the community pool.
Assuming this proposal passes, this is the process we intend to use to liquidate, withdraw, and return any excess funds:
-
Approved ATOM and USDC amounts will be sent from the community pool to a Hypha-controlled wallet.
-
Hypha will use an OTC deal to liquidate the required amount of ATOM (using the 25% buffer to cover price fluctuation and fees if needed) into USDC, resulting in a total of 650k USDC and some remainder amount of ATOM in the Hypha-controlled wallet.
-
Hypha will return all remaining ATOM buffer to the community pool.
- If the ATOM price went down more than 25% over the 2 week voting period, the scope of work may need to be reduced based on the available budget. We will work with the community and the team at Interchain Inc. to get feedback on this.
- If the ATOM price went up over the 2 week voting period, then more than the 25% buffer will be returned to the community pool – Hypha will only sell enough ATOM to cover the amount specified in the funding prop.
-
Hypha will publish a full description of the deal, including documentation of:
- Trading venue for the OTC deal
- Price received in the OTC deal
- Any trading or other fees involved
-
On a monthly basis throughout 2025, Hypha will withdraw funding from the wallet and report the withdrawals to the community.
Oversight and accountability
For the past year, we’ve had quarterly meetings with a six-person oversight committee that was empowered to give strategic guidance and hold both Hypha and Informal accountable for delivering on our promised work.
Our experience was that this format could be very valuable for R&D and product work, but maintenance and operational work was not a focus except when there were major interruptions to business as usual. When it comes to maintenance, no news is good news.
ATOM holders always have ultimate power over teams funded by the Hub, and our commitment to transparency and accountability is:
- We will continue to post quarterly reports on our work in the same format (highlights, challenges, operational report, plans for next quarter, feedback and responses).
- We will continue to post quarterly withdrawal reports on community-disbursed funds. Note that because this is a service-based agreement, there will be no returns being processed.
- We will open an anonymous feedback form and publicly review and incorporate feedback each quarter.
- Our funding will be held in Hypha-controlled wallet. If Hub governance puts forward a proposal to cancel our funding, we will cooperate with any clawback procedures necessary. There is no tooling that currently allows this automatically so for now, that means just returning unused money ourselves.
- If we are removed from our work on the Hub, we will cooperate with whichever team the community wishes to take over.
- If an accountability system is approved for all work on the Hub, we will work within that system to the best of our ability.
Conclusion
To continue our mandate of stewarding the Hub’s testnets, coordinating mainnet operations, and providing development support to the Hub dev team, we are asking for a budget of 650k USD. We commit to public accountability by making quarterly reports on our work, publishing an anonymous feedback form, and responding to relevant feedback in our reports.
We plan on putting this proposal on-chain on January 09, 2024.
Our current mandate officially ends on December 31, 2024 but we will continue working optimistically throughout January 2025 to complete the work planned with the outgoing and incoming Hub teams at Informal Systems and Interchain Inc.
