[Proposal #839][ACCEPTED] Fund 2024 Hub development by Informal Systems and Hypha Worker Co-op


  • 2023-Sep-27: Posted initial draft
  • 2023-Oct-13: Adjusted the naming of opt-in security research topics
  • 2023-Oct-18:
    • Indicated ‘go to market’ status for IBC Routing and Mesh Security as part of the R&D budget.
    • Incorporated further details about headcount in the Budget section
  • 2023-Oct-20:
    • Added performance metrics
    • $5.7M budget will be held in 30% ATOM, 70% USDC
    • No bonus will be paid out unless teams exceed expectations for the quarter
    • Any bonus earned will be placed into a vesting account with a one year lock
    • Added specificity to Hypha’s headcount
  • 2023-Oct-20: Flipped to last-call with intent to go on-chain on October 26. Fixed internal links.
  • 2023-Oct-25:
    • Defined our 3rd party module maintenance responsibilities.
    • Defined our marketing and communications work for the Hub.
  • 2023-Oct-26:
    • Added ATOM numbers to prepare for going on-chain
    • Corrected some mentions of ‘opt-in security’ to ‘partial set security’
    • Added detailed liquidation steps
    • Added Jim Parillo to the oversight committee, replacing ‘representative from Imperator’
    • Updated performance metrics for Interchain Security
    • Added oversight committee names and liquidation multisig names to on-chain summary text

Table of contents


This is a proposal to fund a core development and testing package on the Cosmos Hub for 2024. The teams that would receive funding if this proposal passes are the Cosmos Hub Teams from Informal Systems and Hypha Worker Co-operative. The total budget is $5.7 million USD (to be held in 30% ATOM, 70% USDC), plus 100k ATOM in performance bonuses, for a total of 914,285.71 using a spot price of 7 USD. This funding would replace both teams’ current funding from the Interchain Foundation and have these teams be directly accountable to the community. As of October 26, 2023, this represents approximately 18% of the community pool.


  • ATOM values have been calculated using a spot price of $7 USD as of October 26, 2023.
  • This budget only covers the teams at Hypha and Informal that work specifically on the Cosmos Hub.
  • The ATOM total requested from the community pool includes a 25% buffer on the budget amount to cover fluctuation in ATOM price over the 2-week voting period. Any unused buffer will be returned to the community pool.

This proposal is summarized by the following propositions:

  • The release of 1,117,857.14 ATOM to finance:
    • An overall budget of $5.7M USD plus 100k ATOM in possible performance bonuses, with a 25% buffer to cover price fluctuations between start of voting and liquidation to USDC. Any unused buffer will be returned to the community pool.
    • Maintenance and development of the Interchain Security protocol
    • Maintenance of the Cosmos Hub Gaia core software
    • Maintenance and improvements to the testnet program for the Cosmos Hub and its consumer chains
    • Research and development for improvements to Interchain Security such as Partial Set Security, Mesh Security (go to market), Atomic IBC, and IBC routing (go to market)
  • Thes ATOM will be released to a liquidation multisig responsible for converting the appropriate portion of ATOM into USDC via the process laid out in the full text of the proposal. The liquidation multisig is made of:
    • Simply Staking
    • Stakecito
    • Citadel One
    • CryptoCrew
    • Stakin
  • The ratification of a committee that will oversee the work of Informal and Hypha within the time period of January 1, 2024 - December 31, 2024 and provide insight and visibility to the community about that work. The committee is made of:
    • Stride contributor: Aidan Salzmann
    • Neutron contributor: Avril Dutheil
    • Polkachu representative: Polkachu
    • Jim Parillo at Figment Capital
    • Shane Vitarana at Stargaze
    • James Hinck, Product Manager at Circle
  • An optimistic vesting mechanism which gives the community the power to suspend either team’s work agreement at any time via a governance proposal if that team is not fulfilling their commitment to the Hub

Details of disbursal of funds:

  • The split between Informal and Hypha is as follows:
    • Informal budget: 3.16M USDC + $1.34M worth of ATOM
    • Informal bonus: 79k ATOM
    • Hypha budget: 840k USDC + $340k worth of ATOM
    • Hypha bonus: 21k ATOM
    • Of the budget portions, any unused funds will be returned on a quarterly basis.
    • Any unearned bonuses will be returned on a quarterly basis. Earned bonuses will go into a one year locked vesting account.

Governance votes and outcomes

The following items summarize the voting options and what it means for this proposal:


You agree to fund the Informal Systems Hub team and Hypha Worker Co-operative with $5.7 million USD + 100k ATOM from the Community Pool for the Cosmos Hub’s continued development, maintenance and testing over the period of January 1, 2024 - December 31, 2024, held accountable by their specified oversight committee.

A ‘YES’ outcome will immediately release 1,117,857.14 ATOM to the specified wallet.


You do not agree to fund the Informal Systems Hub team and Hypha Worker Co-operative based on the terms of this proposal.

A ‘NO’ or ‘NO WITH VETO’ outcome will not fund the Informal Systems Hub team and Hypha Worker Co-operative from the Cosmos Hub community pool.


A ‘NoWithVeto’ vote indicates a proposal either (1) is deemed to be spam, i.e., irrelevant to Cosmos Hub, (2) disproportionately infringes on minority interests, or (3) violates or encourages violation of the rules of engagement as currently set out by Cosmos Hub governance. If the number of ‘NoWithVeto’ votes is greater than a third of total votes, the proposal is rejected and the deposits are burned.


You wish to contribute to the quorum but you formally decline to vote either for or against the proposal.


:arrow_up: Back to top

The Hub’s Past and Present: a multi-stakeholder effort

Both Informal Systems and Hypha Worker Co-operative have played critical roles in the go-to-market strategy of Replicated Security, which is the core product offering of the Cosmos Hub.

The Cosmos Hub team at Informal Systems played a key role in the design and development of Interchain Security, and as of January 2023 also took on many duties related to maintaining the Cosmos Hub software and preparing new releases. Hypha has supplemented this work via thorough testing for all releases, operating the Replicated Security testnets program, and providing critical support to incoming consumer chains. Here are some of the teams’ past accomplishments:

  • Designed, developed, and tested the Replicated Security protocol
  • Maintained a steady cadence of smooth Cosmos Hub releases
  • Did business development to onboard a top tier set of consumer chains
  • Did cutting-edge research on shared security and liquid staking technology and economics
  • Developed innovative new simulation and testing techniques to ensure excellent security and uptime

See a more detailed list in appendix A.

Together, these independently operating organizations touch every step in the software development and release process on the Cosmos Hub, which has resulted in the on-boarding of top-tier consumer chains such as Neutron and Stride, with many more waiting to join.

Software released by Informal Systems and tested by Hypha has enabled the Cosmos Hub to maintain a regular cadence of upgrades and stay up to date with the latest versions of its dependencies. Informal and Hypha have also led the emergency release process, smoothly bringing critical upgrades to the validator set when necessary.

The budget in this proposal will allow the Hub Teams at Informal Systems & Hypha to continue their work maintaining the Cosmos Hub, improving Interchain Security, on-boarding consumer chains, and spearheading the development and testing of new features.

Becoming accountable directly to the Cosmos Hub community

During 2023 and before, development of the Cosmos Hub was funded primarily by the ICF. The Informal Hub Team’s budget in 2023 was around $4.5 million, while Hypha’s was approximately $1.2 million.

Funding development of the Cosmos Hub from its own community pool would mark a milestone of decentralization and sustainability, and would allow the Hub development team to be directly accountable to the Cosmos Hub, rather than to the ICF. Maintaining independence from the ICF enables the Hub to set its own standards for innovation, execution, and accountability.

With four years of governance experience, there is no entity better equipped to be directly responsible for funding the development of the Cosmos Hub than the Cosmos Hub community itself.


:arrow_up: Back to top

There are many exciting developments for the Cosmos Hub in progress, but the Informal and Hypha teams have a very clear focus. We work on the lowest level of the stack on the shared security, atomic composability, and routing technologies which underpin the Atom Economic Zone. Our roadmap for 2024 focuses on:

  • Continuing to maintain the Cosmos Hub, cutting releases, working with teams to release new features, etc.
  • Continuing to perfect Interchain Security to make the protocol more robust, sustainable, and scalable for validators to operate.
  • Creating procedures and software to ensure reliable and well tested Cosmos Hub and consumer chain releases, by using simulations and several different kinds of testnets.
  • Research and development to expand the market for existing features and bring new features to the Cosmos Hub
    • Research Partial Set Security to reduce costs on the validator set, and allow a greater number of consumer chains to join.
    • Research Atomic IBC to bring lower cost operation and atomic composability to Cosmos Hub consumer chains.
    • Assistance on Mesh Security development, deployment on the Hub, go to market, and research on complementary features to give the Hub a new way to share its security and even consume security.
    • Go to market and deployment for ICS33 IBC routing to allow Cosmos chains to route IBC packets transparently over the Hub to reduce IBC costs throughout the ecosystem.

Steward the Cosmos Hub

:arrow_up: Back to top

The Informal and Hypha Hub Teams will continue to contribute to stewardship of the Cosmos Hub’s core software, Gaia. This is obviously of core importance to everything else on this roadmap, and the team will continue to uphold the high standards set in 2023.

Enable collaboration for the Hub across the ecosystem

Many different features for the Cosmos Hub are being planned and developed by teams around the ecosystem. The Informal Hub Team has played a key role over the past year in advising, reviewing, and integrating features developed by other teams. Taking the Liquid Staking Module as an example:

  • Analysis: We did extensive analysis and research on the possible risks of liquid staking.
  • Product: We then advised the Stride and Iqlusion teams on the best ways to reduce these risks, informing the addition of features such as the global cap on liquid staking.
  • Organizational: We worked with the Cosmos-SDK, Stride, and Iqlusion teams to decide which repository the LSM code would be housed in, and who would maintain it.
  • Code review: We helped review the LSM code before its deployment on the Cosmos Hub.

We will continue providing this level of support and review for other new features.

Regular release schedule

Maintaining a regular release schedule for the Cosmos Hub has been core to our approach in the past year. Regular releases reduce the risk of any given release, build “muscle memory” for releases in the team and the validator set, and ensure that features are deployed on a timely and regular basis.

Validator support

Informal and Hypha are on-call during both planned and emergency Hub upgrade deployments. We communicate with validators about the software, as well as troubleshoot any issues that come up. Due to this proactive management, Hub upgrades have been very smooth, and have resulted in very little downtime. This was even the case for the one emergency upgrade we have had this year, which resulted in only around 5 minutes of downtime. The smoothness of the actual upgrades is the result of weeks of preparation by our team and proactive communication with a large number of validator teams.

Review dependencies and update the Hub

We are keeping the Cosmos Hub up to date with the latest versions of Cosmos-SDK, IBC, and other dependencies. This also includes deciding when a dependency is ready for deployment, given the Hub’s high standard of security. For example, we updated the Hub to SDK 47, as well as spearheaded the effort to get a 3rd party audit on critical parts of the update.

Marketing and communications

Marketing activities include and are limited to communications around the work and development on the Hub done by Hypha and Informal Systems within the scope of this proposal. Some of these items include Community Developer Calls, episodes of Informal Spaces, Monthly Updates and all of the supporting marketing activities around each of these initiatives. Informal Systems marketing will assist in the marketing of the Hub itself and the expansion of business activities within the scope of the collaborative initiative laid out in this tweet.

Interchain Security

:arrow_up: Back to top

We will continue to perfect and maintain ICS, which is the cornerstone of the AEZ. Our improvements will focus on increasing reliability and performance of the protocol

General maintenance

ICS must work with several versions of IBC and Cosmos-SDK, since different consumer chains use different versions of these libraries. Maintaining releases compatible with these versions and porting features from one to another is not a small task but is critical to making sure that all consumer chains run smoothly.


The consumer chain validator set is currently updated every block. This is best for reasoning about security, but it generates a lot of traffic. We’d like to reduce this to a much lower frequency, for example once an hour. We have to prove that this is secure, but once this is done it will provide a more efficient implementation of ICS.

Read-only protocol

We would like to simplify the ICS protocol to work with IBC queries. This will allow for a much simpler protocol, and easier maintenance going forward, as well as making the Hub more robust to malfunctioning consumer chains. The read-only protocol will also make it much easier to implement features such as opt-in security, and to move faster in general.


We maintain the documentation at https://cosmos.github.io/interchain-security/, which provides a technical and theoretical overview, as well as a guide for validators and consumer chain teams.


:arrow_up: Back to top

The cultures of Informal and Hypha are rooted in software verification and QA, and we put a lot of emphasis on testing in our work on the Cosmos Hub. Beyond our extensive test suites for Interchain Security, we also write some of our own tooling.


CometMock is a stand-in for Comet that can be used during testing which greatly shortens test runtime. Many Cosmos projects have end to end tests that run for 20-30 minutes, and CometMock can reduce this to 1-2 minutes. CometMock also provides determinism (tests run the same exact way every single time), and complete control over time and block production. We built CometMock for use on the Hub, but we are working to bring it to a broader audience of Cosmos projects.

Simulating Cosmos Hub state for testing

Hypha has developed a testing framework to export and modify the Cosmos Hub state. We use this framework to conduct rigorous tests on specific Gaia branches, simulating upcoming mainnet upgrades as closely as possible. This approach significantly reduces the risk associated with new releases, ensuring a high level of confidence when launching them on the Hub and contributing to the overall safety and security of launches. We will continue to enhance this framework, extending its coverage to critical scenarios and using it to test upcoming Gaia releases.

Three-phased testnet release process

Every Cosmos Hub release undergoes a thorough testing process across three phases of testnets. In the development phase, release testnets are automatically deployed using Hypha’s continuous integration scripts. Upon completion of the first release candidate, we conduct local testnets and execute a predefined set of tests. Finally, we upgrade two public testnets with Cosmos Hub validators, enabling extensive community testing. The public replicated security testnet supports multiple consumer chain testnets, facilitating integration testing in a production-like environment. Hypha will continue to oversee the Cosmos Hub and consumer chain releases through this testnet process while maintaining the necessary infrastructure and tools for public testnets.

Testnet Wednesdays for increased validator engagement

Hypha has matured Cosmos Hub’s testnet program, conducting over 50 testnet events in the current year. These events serve as valuable training grounds for validators, software testing, and the identification of operational challenges related to the Gaia codebase. In 2023, we introduced “Testnet Wednesdays,” an initiative aimed at regularizing public testnet events. This initiative allows validators to allocate dedicated time for participation, thereby boosting validator engagement. In 2024, our plan is to further refine Testnet Wednesdays by diversifying event types to cover various testing aspects such as performance, security, and functionality.

Chaos testing for scalability and resilience

As part of routine testing on both local and participatory testnets, we intend to introduce chaos testing patterns. This approach will help us explore operational limits, discover scale-related vulnerabilities, and identify system bottlenecks.

Research and development

:arrow_up: Back to top

The items in this section of the roadmap are still under active research and are more likely to be substantially modified, added to, or deprioritized. Over the course of 2024, we will focus on one or two of these topics to begin building and putting into production.

Opt-in Security

Opt-in security is a family of techniques which could allow fewer validators than the whole set to validate a given consumer chain. This has potential cost and performance benefits, increasing the flexibility of the protocol, while still being a more “full-service” offering than Mesh Security. We will research techniques with which this can be accomplished, and use them to design a protocol that can benefit new and existing consumer chains. There are two basic techniques we are looking at to accomplish this:

Opt-In with Fraud Proofs

In the case where only the stake of the opted-in validators is used to secure the consumer chain, we encounter the subset problem (discussed on the Informal blog here). To be secure it needs something to prevent incorrect execution. This can be done by fraud proofs, validity proofs, or fraud votes. This requirement also exists for mesh security.

Partial Set Security

This research idea has entered the CHIPs discussion phase on the forum. Validators who are not opted-in need to delegate their stake to validators who are opted-in. This allows the full stake of the Hub to secure consumer chains, as well eliminating the need for fraud proofs, validity proofs, or fraud votes. This idea has previously been discussed under names like ‘partial opt-in’, ‘proxy opt-in’, or ‘subset security’.

Atomic IBC

Atomic IBC is a recently released plan that sets the Cosmos Hub up for long-term success. It does this by combining the scalability and sovereignty of the multichain ecosystem with the atomic composability of a smart contract platform like Ethereum. This combination of scalability and composability has long been a goal of many blockchain designers, and Atomic IBC solves it elegantly.

Atomic IBC will give participating consumer chains the ability to integrate seamlessly by composing multi-chain IBC interactions with many steps and delays into one transaction, called an “atomic bundle” which executes across many chains in one step. This can reduce the code required to implement complicated multi-chain workflows like those used by Timewave by around 90%.

One way to make Atomic IBC happen is by having consumer chains share a blockchain with bigger blocks but parallel execution of most transactions. We’re calling this the “Megablocks” architecture. The Megablocks architecture is relatively simple, will provide the best user experience possible, and will reduce the cost of running consumer chains using this architecture.

However, Megablocks limits the amount of customizations that consumer chain teams can make to their low-level Comet configuration and code, and necessitates a shared mempool. For this reason, we are also researching other techniques to enable Atomic IBC, from synchronizing block production across several Comet processes, to heterogeneous Paxos, a consensus technique which lets chains make shared blocks containing atomic transactions when required.

Consensus research

We will work towards building a proof of concept of the Megablocks architecture, while also exploring other techniques such as block synchronization and heterogenous Paxos to enable greater consensus protocol customization. We will also work to rigorously define how Atomic IBC will interact with the low level consensus customizations offered by ABCI++, and make sure that UX benefits offered by Atomic IBC are worth it.

Developer and user experience research

We will explore what the developer interface to Atomic IBC could look like, and how it could make existing and future multi-chain composition use cases easier and better. We will work with existing and prospective consumer chains to see how these UX improvements can benefit their applications.

Protocol design

We will work on specifying and building the Atomic IBC protocol. This will entail changes to IBC to allow it to pass messages sequentially between Atomic IBC consumer chains as part of an atomic bundle, and calculate gas costs for atomic bundles. This will also entail changes to Comet to allow it to roll back transactions in an atomic bundle if any of the other transactions error.

Mesh Security (go to market)

The Informal Hub Team has lent our expertise to the Mesh Security team, on topics such as slashing and fraud proofs. We will continue to collaborate with them to further the development and launch of Mesh Security and share concepts between the protocols to ensure that the Cosmos Hub can be well positioned for a mesh-security future.

IBC routing (go to market)

One of the Cosmos Hub’s original missions is to route IBC packets between other Cosmos chains to increase the efficiency of the interchain by reducing relaying and light client update costs. This has been delayed by the lack of working routing in IBC. Recently, a workable IBC routing scheme has been proposed by the team at Polymer. The scheme allows IBC packets to be routed over the Hub without requiring any data to be written to the Hub, by aggregating IBC light client proofs from many connected chains.

This will augment the packet forward middleware already installed on the Hub by providing transparent routing – applications will have to make no distinction between direct and routed packets. For example, tokens transferred with this style of routing will be indistinguishable from, and fully fungible with direct routed tokens.

The Informal Hub Team will lead an effort to get this deployed to the Cosmos Hub, the Hub’s ICS consumer chains (including those using Atomic IBC), and offer it to other chains in the Cosmos ecosystem.

We will consult with the Polymer and IBC-Go teams to find the best path to deploying this feature. We will also help implement any additional code that is required. We will support existing ICS consumer chains in deploying this feature to reduce their relaying costs. We will also work to educate the broader community about it and introduce the feature to non-consumer chains.


:arrow_up: Back to top

Performance metrics

Research and development

Goal: Bring innovative features to the Hub

Suggested metrics:

  • Proposals progressing through CHIPs process
  • New features included in Gaia releases

Interchain Security

Goal: Make the Hub an appealing and sustainable provider chain

Suggested metrics:

  • Cost reductions for operators measured in number of consumer chains
  • The Hub being an attractive provider chain, also measured in number of consumer chains
  • Quality metrics of consumer chains (e.g., TVL, market cap, social media sentiment)

Gaia maintenance

Goal: Keep the Hub up-to-date and running smoothly

Suggested metrics:

  • Consistent releases over the funded year (est. 2 releases per quarter)
  • Inclusion of core stack version updates in Gaia releases, pending stack version releases (est. 2 sdk updates, 2 ibc updates)
  • Time per upgrade

Testing and testnets

Goal: Ship bug-free code to mainnet and improve operational efficiency

Suggested metrics:

  • Security and liveness issues addressed using the testnets
  • Consistent testnet events (upgrades, launches, wargaming sessions)
  • Testnet participation measured in validator participation
  • Quality of learnings measured in testnet reports 1

Oversight committee

This committee exists to oversee, advise, and guide the core teams and the community in regards to the Informal Systems and Hypha Co-op Hub teams’ work throughout their fulfillment of their Cosmos Hub development, maintenance, and operational work related to this proposal.

This committee is ratified by the passage of this funding proposal. The committee represents various stakeholder groups in the community that are equipped to perform oversight to these teams, are impacted by their work, or able to provide an outside perspective. It includes representatives from consumer chain teams (representing the Hub’s “customer” voice), Hub validators (representing the Hub’s “operator” voice), and prominent projects in the industry who can provide an external point of view.

The committee’s power ultimately comes from the confidence and support of ATOM token holders. If, at any time, the token holders believe that the committee is not providing sufficient oversight or that either team is not fulfilling their commitment to developing the Hub, a governance proposal can be passed to dissolve the committee, suspend funding for either or both teams, and return unused funds to the Community Pool.

With these things in mind, the committee will:

  • provide oversight to the Hub teams at Informal Systems and Hypha Worker Co-operative in their capacity as core teams of the Cosmos Hub for the funding period of January 2024 through December 2024.
  • assess work done by the Hub teams at Informal Systems and Hypha Worker Co-operative. This assessment will be based on reports written by Hypha and Informal (see appendix B) and will result in grades submitted after each quarterly meeting (see appendix C). These reports and grades will be made available to the community.

The committee will not:

  • act as a body overseeing any Hub development aside from that of Informal and Hypha. Other community pool funded efforts are out of scope, and may have their own committees or other governance structures.
  • act as a lasting governing body. The committee is effective for the duration of the proposal (from January 1, 2024 - December 31, 2024). The committee, together with the core teams, are put in place by governance approving this proposal.
  • make final decisions on strategic direction or roadmap. These decisions should be guided by Informal and Hypha and ultimately approved by the Hub’s community and customers.

Responsibilities and Output

The oversight committee will convene for 5 meetings during the 2024 year, with one meeting at the beginning of the year, and one meeting at the end of each quarter. Each meeting will consist of presentations by representatives of Informal and Hypha, followed by questions and discussion from committee members. Committee members and core team members will communicate asynchronously to iterate on feedback given during these quarterly meetings.

Meeting minutes and summaries will be produced by the core teams and made available to the community after being approved by the committee along with the teams’ quarterly reports, and the committee’s grades on the quarter’s progress. The format of the teams’ reports and the grading system are shown in appendix B and appendix C.

2024 roadmap meeting

Occurs at the beginning of the year (January 2024) to discuss and formalize roadmaps for each team.

  • Presentation on 2024 product agenda
  • Presentation on 2024 operational and testing goals
  • Questions and discussion from committee members

Quarterly assessment meetings

Meeting at the end of each quarter (i.e., Mar 2024, Jun 2024, Sep 2024, Dec 2024) to assess road map progress and undertake the review & reporting process.

  • Presentation from Hypha and Informal on the quarterly report (see appendix B: Quarterly reporting format)
  • Questions and discussion from committee members
  • Within a week: Submission of grades by committee members, signoff by committee on meeting summary, and posting of quarterly report, meeting summary and grades on forum.


The membership of the committee will not change throughout the funding period named in this proposal. The members of the committee are as follows:

  • Stride contributor: Aidan Salzmann
  • Neutron contributor: Avril Dutheil
  • Polkachu representative: Polkachu
  • Jim Parillo at Figment Capital
  • Shane Vitarana at Stargaze
  • James Hinck, Product Manager at Circle

Procedures around team removal

A core part of the committee and the community’s oversight over the team rests on their ability to remove the team and end funding. However, this should take place in a smooth framework so that the team has the ability to correct performance issues if any exist, before the “nuclear option” is invoked.

If, for two consecutive quarters, the committee submits a grade of “Needs improvement” in the overall performance category, they must also write and submit a signaling proposal calling for removal of the team and suspension of the funding.

Hypha and Informal submit reports separately, are graded separately, and can be removed separately.

As a reminder, any member of the community can submit a proposal to remove funding at any time, without following this procedure.


:arrow_up: Back to top

Our total budget for 2024 is $5.7 million, which will be held in 30% ATOM, 70% USDC. This budget covers the following broad areas listed below, and this is roughly how we expect to distribute effort within these areas. While these line items are not exact, they represent the approximate time investment from our teams on each of these areas.

Budget Notes
R&D $1,700,000.00 Partial Set Security, Atomic IBC, Mesh Security, IBC Routing
Interchain Security $1,500,000.00 Read-only protocol, epochs, general maintenance and support
Gaia maintenance $1,500,000.00 Upgrades, release management & incident response
Testing and testnets $1,000,000.00 Simulated testing, Cometmock, testnets validator outreach
Total $5,700,000.00

This request aligns closely with the 2023 funding allocated by the ICF, maintaining a consistent team size. The rates charged are in line with standard market rates and encompass various costs such as salaries, operational expenses, and team incentives in ATOM.

This budget is crucial to support the growth and competitiveness of the Cosmos Hub, while ensuring ongoing motivation and dedication from our team members. The total Hub team’s headcount will typically range from 14 to 20 individuals (2-4 for Hypha, 12-16 for Informal) with varying levels of experience, subject to occasional fluctuations due to turnover and hiring cycles. On a quarterly basis, both Hypha and Informal will provide reports outlining the amount of funds used to date for team members directly involved in the project (see appendix B). Rates are aligned with the mid-point of mid-market software development rates which typically range from $110-$220/hour.


The Informal Hub Team budgets for 14 team members. This includes:

  • A project lead, product owner, and operations manager, who oversee and participate in every part of the team’s work.
  • 5 generalist software engineers, who write most of the code.
  • A protocol designer, who answers complicated questions about protocol correctness
  • A verification engineer who builds and uses automated testing technology like CometMock, Quint, and Model-based testing
  • A finance/economics researcher who answers quantitative questions about the financial impacts of technologies (like liquid staking)
  • A devops engineer who helps Hypha with production operations and works on performance related aspects of the code and infrastructure
  • 1 FTE equivalent for marketing and BD support, to help communicate about the Hub and our work, and find and guide prospective consumer chains.

The Hypha Hub Team’s budget includes:

  • 2 general software engineers who write code, design and build tests, and work with other dev teams
  • 1 product manager who manages scope, timeline, communications, and is involved with every part of our team’s work
  • 0.5 devops engineer who manages and orchestrates our fleet of nodes on two public testnets and multiple consumer chains

Third party module maintenance
For Hypha, moving into Gaia maintenance and release integration would involve increasing our headcount to 4. Hypha plans to allocate 1 FTE towards Gaia maintenance. Of this, 0.5 time will be dedicated to release integration and cutting release candidates. Remaining 0.5 time will be dedicated to maintenance of governance approved Cosmos Hub modules developed by third party developers (not by Informal or Hypha) which must be integrated into the Gaia binary. We encourage developers to submit feature ideas and code for integration into Gaia, preferably through the CHIPs process. Although our recommendation is that third-party developers maintain their modules, we have budgeted dedicated time to support their maintenance. This allocated time will ensure we can perform routine upkeep and manage module dependencies promptly. However, if the maintenance requirements of third-party modules exceed our budget, it may lead to release delays or a deprioritization of other tasks as we reassign developers.

Per-head rate
For both organizations, our per-head total rate averages around $325K per year, this is how with Informal’s 14 and Hypha’s 3.5 team members, we get to a cost of $5.7M. This total cost includes approximately $225K in salary (and relevant overhead such as employer taxes and benefits), the remainder includes standard overhead margin to pay for operational expenses like legal, HR, finance and other support as well as some profit margin to the companies, to ensure long term sustainable growth.

We will include details on the number of employees working on the Hub in our quarterly reports. In case we fall below the allocated number of employees during a quarter (due to turnover etc.), we will only use funding from the multisig needed for the number of employees actually employed.

Performance bonuses

The 100k ATOM performance bonus will be awarded quarterly (up to 25k per quarter) based on the “Overall performance” grading category (see appendix C), which is determined based on several factors including KPIs set in the previous quarter.

  • “Needs improvement”: 0% of bonus awarded
  • “Meets expectations”: 0% of bonus awarded
  • “Exceeds expectations”: 100% of bonus awarded

If a quarterly performance bonus is earned, it will be withdrawn from the yearlong vesting accounts by each team and immediately transferred to a vesting account with a one year lock on it to further incentivize good performance from the teams. If the performance bonus is not earned, it will be immediately returned to the community pool.

Funding Control Mechanism

Implementation using classic Cosmos-SDK vesting multisig

This implementation involves several compromises because of the inflexibility of the classic SDK 0.45 vesting multisig currently running on the Cosmos Hub.

  • Vesting schedule: To approximate payments at the beginning of the month, vesting will start one month early, in December 2023. This means that the full amount necessary to meet payroll for the month will be available to the core teams by the beginning of January 2024, and every month thereafter, with full vesting complete at the beginning of December 2024.
  • Multisig control: Each funded team will control their own multisig vesting wallet. Core team representatives will only be able to withdraw funds that have already vested. Withdrawals occur on a monthly basis.
  • Dissolution: To end funding for a particular team, an upgrade proposal containing migration code moving the funds out of the vesting multisig and back to the community pool must be passed via governance on the Cosmos Hub. It is possible for anyone to introduce this proposal, but doing so is an explicit part of the mandate of the committee.

Liquidation (step by step process)

  1. If/when proposal passes: Total ATOM amount will be sent from community pool to a 3 of 5 interim multisig comprising:
    • CryptoCrew
    • SimplyStaking
    • Stakecito
    • Stakin
    • Citadel One
  2. The interim multisig will use an OTC deal or swap to liquidate $4M worth of ATOM (using the 25% buffer to cover price fluctuation and fees if needed) into USDC. This $4M represents 70% of the total budget. The remaining 30% will be retained in ATOM. The end result of this step is that 5.7M USD worth of tokens (70% USDC, 30% ATOM) will be made available to send to the teams.
  3. The interim multisig will send the following transactions to vesting accounts managed by the respective teams. While all accounts are continuously vesting over 2024, the ATOM budget and ATOM portions are divided for ease of transparency and reporting.
    • Informal budget: 3.16M USDC + $1.34M worth of ATOM
    • Informal bonus: 79k ATOM
    • Hypha budget: 840k USDC + $340k worth of ATOM
    • Hypha bonus: 21k ATOM
  4. The interim multisig will return all remaining ATOM buffer to the community pool.
    • If the ATOM price is exactly the same as when the prop went up for voting (2 weeks before this point), then the entire 25% buffer will be returned to the community pool.
    • If the ATOM price went down less than 25% over the 2 week voting period, then some of the buffer will have been used to cover the difference, and less will be returned.
    • If the ATOM price went down more than 25% over the 2 week voting period, the scope of work and/or headcount may need to be reduced based on the available budget. We will work with the community and our oversight committee to get feedback on this.
    • If the ATOM price went up over the 2 week voting period, then more than the 25% buffer will be returned to the community pool – the interim multisig will only sell enough ATOM to cover the amount specified in the funding prop.
  5. The interim multisig will publish a full description of the deal, including documentation of:
    • Trading venue for the OTC deal
    • Price received in the OTC deal (it may be necessary to offer a slight discount from ATOM’s spot price)
    • Any trading or other fees involved

Appendix A: Past work

:arrow_up: Back to top

Informal Systems came up with the original protocol design and specification for Interchain Security in 2021, and led the implementation of Replicated Security shortly thereafter. Replicated Security is a complex piece of software which includes the core provider and consumer modules, a protocol to allow consumer chains to have their own unbonding periods, key assignment, and the sovereign-to-consumer migration feature which allows a transition to Replicated Security without disrupting IBC connections.

Interchain Security has become the Cosmos Hub’s flagship product and we continue to lead its development through six releases of Gaia (v8 through v12) this year. Informal has also published research on Shared Security and comparisons of different security options for the Hub.

Together, Hypha and Informal have implemented several testing frameworks for Replicated Security, making QA and verification much easier for builders using this technology.

In 2022, Hypha re-established the Cosmos Hub Testnet program and has since built it into a thriving testing ground for every consumer chain on the Hub. We coordinated Cosmos Hub’s third incentivized testnet, _Game of Chain_s, and continued that work into the Replicated Security Testnet. This testnet has been the home of ten launch rehearsals, four synchronized upgrades, and one re-launch; it is an instrumental part of the onboarding process for Hub consumer chains.

Hypha has continued to publish educational content and analysis of Replicated Security (Swiss Booklet: Neutron, Preparing for Replicated Security, Conditional Basic Income) throughout 2023.

For a detailed breakdown of our teams’ past work, see here this document.

Appendix B: Quarterly reporting format

Click to open Appendix B: Quarterly reporting format

:arrow_up: Back to top

Hypha and Informal will submit and present separate quarterly reports, and will be graded separately.

Work done in previous quarter

This is an overview of work done by the team in the previous quarter, and a review of how it matched up to the plan for the quarter.


Completed tasks and successes.


Tasks that took longer than expected, or turned out to be more complicated. Unexpected events such as security incidents, etc.


Places where we deviated from the quarter’s plan, due to better understanding of tasks involved, unforeseen circumstances, and strategic adjustments.

Plan for next quarter

Taking into account the work and events of the previous quarter, what is planned for the next quarter? This is also guided by the overall roadmap and long term strategic goals from the roadmap.

Major areas of work

What are the major focuses this quarter? What are some individual tasks or deliverables that will be worked on? What KPIs will this work be judged on?

Expected challenges

Some tasks involve a higher level of uncertainty than others. This is a place to highlight expected complications in the work.

Community input

One of our core roles is to build what the Cosmos Hub community wants. We also take the role of leading through research and development, but ultimately, everything deployed on the Hub is subject to community approval.

Input received

In which ways has community feedback been solicited in the past quarter, and on what? What is the feedback?

Interpretation and incorporation of input

How has this feedback modified the work executed this quarter, next quarter’s plan, and the year’s roadmap?

Operational report

Baseline maintenance and regular processes are a significant part of the teams’ work. Operational tasks are not necessarily dependent on the roadmap or subject to shifting over quarters, but reflect the consistency and reliability of core operations.


What Hub upgrades, emergency or planned, have happened? How did they go?


What consumer chain testnets, launches, and upgrades have happened?

Security events

Were there any security events that happened? What was the outcome and how were they fixed?

Funding report

Burn rate

How much was spent this quarter? (of course, the vesting account puts an upper limit on this)

Remaining funds

How much remains for the year?

Appendix C: Grading

Click to open Appendix C: Grading

:arrow_up: Back to top

The committee will assess the work of the previous quarter in four categories, and give a grade for each one, along with an overall grade. There are three grades:

  • Exceeds expectations - the team is going above and beyond their commitments
  • Meets expectations - the team is performing their job and fulfilling their commitments to the Hub
  • Needs improvement - the team is falling short of their commitments

These are the four categories, which roughly correspond to the areas covered by each report and result in an overall performance grade:

  1. Meeting last quarter’s plan

How well did the team execute against the plan and KPIs set at the last meeting?

  1. Alignment and progress on overall roadmap

How well do last quarter’s execution and this quarter’s plan fulfill the high level roadmap?

  1. Integration of community input

How well did the team do in considering input from the community, and putting processes in place to analyze and integrate this input?

  1. Operational smoothness

How well have testnet, upgrades, and general operation been going on the Cosmos Hub and consumer chains?

Overall performance

How well is the team doing overall? Grades for each of the four categories should factor into this (i.e., If all four categories are ‘needs improvement’ then the overall performance should logically not be ‘exceeds expectations’). Two consecutive quarters of ‘needs improvement’ should result in a signaling proposal to remove the team.


As a contributor to Neutron, I can only attest to the amount and quality of the work both Hypha and Informal have put into the Hub and Replicated Security. I think this proposal marks a step in an interesting direction for the Hub, bringing the core teams, their funding and accountability within the scope of the Hub’s governance.

Refining exactly how that is done, to ensure the financial stability of the core teams, their performance to the hub, and the mechanisms by which the Hub ensures accountability, is likely to take years of experimentations and fine tuning, but I think this proposal is a strong step in the right direction and has clear been thought through.

Personally very supportive.

PS: BORG-ify the Hub :fire:


Hey teams,

First, I’d like to start with overall support of the Hub funding its core development and thorough and rigorous testing. The Hub has relied on the ICF to fund core and critical development for far too long. I’m glad to see someone stepping up on this front.

As for the proposal itself, this feels like an omnibus. While these items may seem related, they each have different scopes and deliverables, and teams work on them. It would be best to have the community vote on each as separate funding for the year, with their own schedules, deliverables and KPIs to be monitored against.

The community may want Maintenance and Testnet but may feel the R&D components are not worth funding from the community pool and may be best funded via ICF (as is their mandate).


Unless the Hub is going to license the software developed by it so it is open-source but not forkable/re-usable by other chains without permission/payment to the Hub, this seems like something that should not be funded by the Hub and should go through the ICF.

The items in this section of the roadmap are still under active research and are more likely to be substantially modified, added to, or deprioritized. Over the course of 2024, we will focus on one or two of these topics to begin building and putting into production.

This doesn’t instil confidence that this is worth funding via the community pool. The proposal should be specific in the items you will build and deliver for the Hub. You’re setting a roadmap of specific items the community would like to see implemented on the Hub, not what you’d like to pick from the list and implement. Another reason this should be funded by the ICF and specific items presented to the community to be implemented on the Hub/AEZ through governance approval.



For the requested budgets, understand each area’s cost breakdown by component and headcount of each team. Informal and Hypha are for-profit businesses. Understanding the costs in far more detail when requesting public funding is not unreasonable.

How much effort or FTEs will be focused on any one item within the items listed in the budget.

To be awarded a bonus, you need to be able to measure and monitor deliverables. Your proposal has extremely loosely defined deliverables. Measured only be sentiment check via the Oversight Committee, this isn’t sufficient.

A few examples of measurable and monitorable KPIs would be:

Report on any Incidents related to the Hub, AEZ or ICS chains within 24 hours of resolution.
Implement 2 Cosmos SDK version updates within the defined proposal timeframe.
Implement 2 ICS version updates within the defined proposal timeframe.
Implement 2 IBC version updates within the defined proposal timeframe.

From here you could measure against these KPIs and award a Bonus.

As for the bonus itself, I’m not in support of this component. You’re completing work for the hub, funded by the community by selling ATOMs to USDC. If you want a bonus, keep a portion of the requested funds in ATOM and be rewarded for the success your work brings to the ATOM token.

Schedule and deliverables

Outside the items above, the proposal lacks any form of schedule and tangible deliverables for the community to monitor against. Here is an example of what I think a roadmap of deliverables for the items identified could look like.


Given the nature of Cosmos governance, you’re not guaranteed to be able to deliver on all of these items, the specific schedule on release,s etc. That is OK. You can put assumptions and constraints into the proposal. “We will implement a minimum of 2 and maximum of 4 Cosmos SDK, IBC version increases. Subject to governance approval.” is perfectly fine to add. At least this gives the community an idea of what you’re targeting to deliver through the year.

Gaia Maintenance

Cosmos Hub already funds Notional through Prop 104 for Gaia Maintenance and Incident Response. Can you please explain how Informals proposal and Notionals proposal differ? Or is this a ‘double spend’ for this area? I can see how the hub could want two teams focusing on maintenance and incident response, I’d just like to understand if there is overlap and if so, how the teams collaborate and complement each other to get the best outcome for the Hub.




No doubt about Hypha and Informal quality, no doubt about the need for Atom community to fund by itself research and dev for Atom.

Where i’m confused again and again is why every single Multisig and now “Oversight Committee” is ~ always composed of the same people?

1 Like

So, on the topic of R&D and performance bonuses:

Research & Development:
First of all, I actually doubt that the ICF is going to want to fund R&D for the Hub. I think they’re making it relatively clear they’re shifting focus to just the core Interchain Stack. Furthermore, it’s pretty common practice to fund R&D for blockchain teams, regardless of whether it’s licensed software. Most of the R&D topics laid out only apply to the Hub anyway.

I’m not actually opposed to the Hub licensing its own software tbh. The big question is what legal entity sits behind this tho.

Performance Bonus:
If we want to compete with other networks, performance bonuses are critical. We need our devs to enjoy the upside of ATOM if they do well. Every other network with an ounce of success in Cosmos is doing this. We don’t want it to be more attractive to work for some other network like Osmosis than it is working for the Hub. Talent retention in this space requires performance bonuses.

You can’t just say “oh don’t convert some of the ATOM then” and expect devs to wanna stick around. Base salary needs to be equally competitive as well.

On the KPIs, I do think you make a good point there @Rarma and it would be good to have some more quantifiable ways to track “success”.

In general I am very supportive of this proposal, but given the high amount of money requested I’d love to see some clarity on the headcount allocation per item. What’s also unclear to me is how many folks on the team are engineers v.s. marketing / BD. The Go-To-Market for Atomic IBC and IBC routing is unclear and I think it would be great to get some clarity on the resources allocated to those two efforts specifically and what the responsibility division is between the ICF’s marketing team that currently runs a lot of the Cosmos Hub’s social media.


Informal Systems and Hypha Worker Co-op funding source should be fully funded by ICF.

Unless the tens of millions of ATOMs managed by ICF are transferred to the COSMOS HUB community pool and managed together.

Several questions:

  1. What will be the role, function, and use of funds of ICF in the future?

  2. Will all COSMOS ecosystem development teams apply for funds from the COSMOS HUB community pool in the future?

After seeing this proposal, I wanted to start from COSMOS
With withdrawals from the HUB community pool, the value of COSMOS HUB will definitely die in the future.

1 Like

Both the Hypha and Informal teams have demonstrated a proven record of delivering cosmos and ICS developments and have effectively stewarded the Cosmos Hub with regular highlights to the community. It is evident that the current iteration of Interchain Security requires enhancements to meet the expectations of all stakeholders, and it is good to see a plan in place to achieve this.

Hopefully, that a significant portion of the Research and Development efforts will transition smoothly into the Development phase, with successful deployment in 2024.

Given that most of the efforts are concentrated on enhancing the Cosmos Hub, it is indeed a prudent decision to establish direct accountability of these teams to the Hub’s community. While @Rarma’s suggestions hold significance and merit further exploration, Everstake, in general, extends its support for this proposal.


To answer question 1: the ICF has been quite clear about what’s in scope for their funding and what isn’t in 2024.

Question 2: Cosmos SDK / IBC / CometBFT teams will (& should) NOT apply for funds from the Hub, unless this regards specific functionality that’s only being built for the Hub. But that’s not the case as of today.


This proposal only works for the Hub so the community pool is fair.
Your comment should used for others who ask for funds for Cosmos.

1 Like

Informal systel has been raising fund 5.3M in Q3 2023 via VC.

Now asking for 5.7M here.

I wonder what will those investor will get and community hub will get too from IS?

Those 2 team will be accountable to cosmos community but how? accountable seems not accurate. We should tell now than later.

Thanks for commenting Rarma. I saw your tweets about this and am glad to see it on the forum as well. You bring up a lot of really good points, and we want to do them justice by writing a detailed response. It will probably take us a couple of days to finish addressing them, but just wanted to let you know we are working on it!


Interchain security and IBC are concepts Cosmos pioneered over 5 years ago that are only now reaching adoption and maturity at a larger scale. Many similar projects have emerged in other ecosystems, some with orders of magnitude more funding than what this proposal is requesting. It makes sense for ATOM stakeholders to directly support the development of these technologies instead of having to seek grants from other associated entities. The Hub’s version of Interchain security / shared security gives the most optionality out of all similar architectures, and thus deserves enough funding for this vision to be realized.

Informal / Hypha have shown to be more than capable in successfully launching Replicated Security, with Neutron and Stride becoming consumer chains. In terms of engineering complexity, it was close to the order of difficulty of Ethereum’s Merge.

The only thing I’d like to see in this prop is more of a breakdown of Informal’s headcount (i.e: what are the roles of the 12-16 individuals).

Looking forward to Atomic IBC and megablocks!


Thank you all for the discussion above.

We are very supportive of the teams’ decisions to approach the Cosmos Hub community pool for funding.

It is a priority for us at the ICF to coordinate funding with other communities and teams because we want to foster a self-sustaining ecosystem. We want to foster effective, sovereign, and decentralized communities.

The teams’ decision to go for CP funding helps accelerate this process – enabling us to refine our focus on the core stack and broader ecosystem. We also believe that the Cosmos Hub is a beneficiary of the growth of the broader ecosystem.

We aim to minimize our impact on the Cosmos Hub over the long-term not because we don’t care about the Cosmos Hub but because we deeply care about sovereignty and autonomy.

With that said, we are open and listening to the community’s feedback about their capacity to fund and support components of core development. We are flexible enough to accommodate the decisions made by the community in line with our strategy of supporting where we are needed to foster a self-sustaining and prosperous ecosystem around the Hub.

The Cosmos Hub is very important to us, and we want to see it thrive.


Thank you Lexa for your continued efforts and dedication to the Hub. Makes total sense for the Hub to funds its own development.

However, would agree with Rarma, Noam and others re the need for greater clarity around KPIs, deliverables and how the funds will be spent.

Going forward, I think it would also be good if we could figure out some relatively straightforward way of allowing the community to have some say in the composition of these committees.


Thanks for the detailed write-up @lexa - there’s no doubt about the importance both Hypha and Informal play within the ecosystem.

I’d like to express that I am in favour of having the funding come from the Community Pool - I think this is an important step and would ensure that the accountability is with the Hub rather than another party.

With respect to the above, I am in agreement with Rarma. I think it would be beneficial to have a separate funding request and possibly different KPIs set for the different teams as they would be working on different aspects of the Roadmap.

This is something that I think needs to be defined further before going on-chain.

Take the example of Hypha who mention they will have 2 - 4 individuals working on the Hub. How many of these are software developers vs. other roles? The rates being quoted were for Software Devs.

Taking the mid-points of what was quoted (3 individuals & $165 per hour) this would amount to roughly $400K per individual which might sound fine but we should have more of a breakdown for transparency’s sake.

Overall I am in favour of the proposal idea, just want to see more transparency regarding salary / operational figures and KPIs.



Gets kinda hard to add any constructive criticism after this post. Well done Rarma. Great feedback


Thanks for commenting here, @Rarma. Your response seems to have captured a lot of what everyone else was wondering too. To address your questions, and the questions of others, I’ll address it point by point:

Omnibus proposal

It’s important to have a core team that works on low level research and development of the Hub’s core features (right now this is ICS), while also integrating efforts from other teams. This is how most chains (and most software projects in general) function. Having a core team doing both maintenance and R&D builds expertise and continuity around the core features. If it’s not us, I would want to see another team filling this joint role.

While these items may seem related, they each have different scopes and deliverables, and teams work on them.

I’ll have to disagree on this a bit. Of the areas of focus we list, I think they are all quite related.

Testing & Testnets and Gaia Maintenance are obviously pretty core to the Cosmos Hub. These categories are also where the Hypha and Informal Hub teams work closely together. We could have submitted separate proposals for each team, but since we work so closely together, and our work would be greatly disrupted if one team’s proposal passed and the other didn’t, we decided that logically they should be combined.

A lot of work on ICS is maintenance, fixing issues in the backlog, improving performance, and upgrading to new SDK versions. There are also protocol improvements, as listed above. We anticipate that this ICS work will also encompass implementing and bringing to production variations on ICS developed through our work in the R&D section.

The tasks in the R&D section are less well-defined, since the nature of research is to find the best path forward. However, this section is also tightly focused on creating core infrastructure for the AEZ. Partial Set Security (an evolution of Opt-in Security) is obvious. The high cost of adding each consumer chain with replicated security slows the rate at which the AEZ grows. Partial Set Security provides a way for fewer validators to run each chain. Atomic IBC provides a way for chains in the AEZ to communicate instantly, while also reducing expenses by having them share blocks.

Mesh security and multihop routing are in the R&D section, but they might also be categorized as “go to market”. These are both technologies that are being developed by other teams (with some help from Informal in both cases), but where the Hub has the potential to be a market leader. Our goal with these is to make sure we are on the leading edge of their deployment, and do business development to get other chains to choose the Hub as their first choice for mesh security and multihop routing.

R&D accountability

This doesn’t instill confidence that this is worth funding via the community pool. The proposal should be specific in the items you will build and deliver for the Hub. You’re setting a roadmap of specific items the community would like to see implemented on the Hub, not what you’d like to pick from the list and implement. Another reason this should be funded by the ICF and specific items presented to the community to be implemented on the Hub/AEZ through governance approval.

It would be best to have the community vote on each as separate funding for the year, with their own schedules, deliverables and KPIs to be monitored against.

Researching and developing new technology is never a sure thing, and it’s impossible to predict with 100% certainty what the right path is. This uncertainty comes from the fact that specifications and prototypes are developed, information is gained which influences the course of subsequent research. We expect that either Atomic IBC or Partial Set Security will become the main focus of new feature development during 2024.

Uncertainty on Atomic IBC comes mostly from questions of whether the proposed underlying “megablocks” architecture will impose unacceptable costs on consumer chains in the form of resource contention, or unacceptable constraints on consumer chain’s use of the ABCI++ protocol. We will answer these questions by building a prototype of megablocks during early 2024, analyzing its performance, and developing solutions to enable the full use of ABCI++. Based on this information, we will either move on to hardening megablocks for production and making the modifications necessary to the IBC library, or we will shift focus to other R&D items.

Partial Set Security does not have as much uncertainty associated with it, since it is a relatively straightforward upgrade to ICS. However, the design space for the feature is quite large and there are still unresolved questions about it. We go over some of the questions in the CHIPs discussion phase post linked above.

Some of the concerns around Partial Set Security, ICS, and even Cosmos dPoS in general can be alleviated by proportional slashing, which slashes accidental double signs much more mildly while slashing real double spend attacks much more harshly. In some sense, research on Partial Set Security and proportional slashing could be seen as a more conservative research path as opposed to spending time on Atomic IBC. We’re open to any comments on where we should be focusing.

Requiring new proposals and votes for each phase of research would be a large amount of overhead, and would create a large amount of employment uncertainty for our engineers in a way that annual funding does not. Additionally, there would be perverse incentives at play. After putting a lot of work into specifying an area of research ahead of time, and hiring engineers for it, we would not be incentivized to shift our efforts to a more fruitful area, since that would require ending the grant and returning remaining money. This would result in dead ends being followed and a lack of flexibility. We do not think that this is a good way to do research.

That being said, we absolutely want to be accountable to the community. We are working on a process for Cosmos Hub feature development called CHIPs. We’ve posted it on the forum here to gather community input on the process. This process draws on Ethereum’s EIP process to ensure that features under development have sufficient community review during the process. The process has 9 phases. Phases 1, 2, and 3 will be taking place under the R&D budget in this proposal, while later phases move into the ICS, Gaia Maintenance, and Testing budgets. I’ll look at the first three here:

  1. Discussion Phase: An idea is proposed and discussed on the Cosmos Hub forum. This proposal can be looser or higher level, but introduces and describes an idea, research direction, or feature. Depending on the outcome of this discussion, authors can evaluate if it is a good idea to put more time into the project.

  2. Specification Phase: A design document is created providing detailed information on the proposed feature. The outcome should be a Cosmos Hub Improvement Proposal (CHIP) following the format defined in https://github.com/cosmos/cips. This document should be posted to the Cosmos Hub forum. At this point, the specification is a first draft. It will likely be changed during the Spike, Signaling, and Implementation phases.

  3. Spike Phase: A prototype implementation of the feature is created that enables better estimation of the expected timeline. This implementation is not expected to be fully finished or hardened for production, but is intended to gain additional information about the feasibility and design of the feature.

    (phases 4-10 are related to production implementation, hardening, and deployment, and not under the R&D budget)

Right now, Atomic IBC and Partial Set Security (an idea for a viable version of Opt-in without fraud proofs) are entering the discussion phase. Here are the forum posts for discussion of Atomic IBC Megablocks research and Partial Set Security. They should be in the specification stage in early 2024. We don’t have it ready yet, but we may also put proportional slashing (slash less if only one validator double signs by accident, like Polkadot and Ethereum) into the discussion phase for a 2024 implementation as well.

Once the discussion phase is complete, the finished specifications and plan for prototypes (phase 2) will also be posted to the forum. From this process, the community and oversight committee should be able to judge whether the R&D work we are doing is worth it. If we are not producing useful research, I’m sure we will get many responses to that effect on the forum. From here we can adjust and course-correct if this is necessary. If we are not course-correcting, or the course we are correcting to is crappy, the committee or community can put forward a proposal to cut our funding, possibly leaving funding for maintenance and testnets intact if that part of the mandate is going well.


I think research into more restrictive licenses for Hub-funded R&D is a great idea, but we are not in the position to suggest a license, as we are not lawyers. However, we are open to working under any license that the community approves by a signaling proposal.


The Informal Hub Team budgets for 14 team members. This includes:

  • A project lead, product owner, and operations manager, who oversee and participate in every part of the team’s work.
  • 5 generalist software engineers, who write most of the code.
  • A protocol designer, who answers complicated questions about protocol correctness
  • A verification engineer who builds and uses automated testing technology like CometMock, Quint, and Model-based testing
  • A finance/economics researcher who answers quantitative questions about the financial impacts of technologies (like liquid staking)
  • A devops engineer who helps Hypha with production operations and works on performance related aspects of the code and infrastructure
  • 1 FTE equivalent for marketing and BD support, to help communicate about the Hub and our work, and find and guide prospective consumer chains.

Our per-head total rate averages around $325K per year, this is how with Informal’s 14 and Hypha’s 3.5 team members, we get to a cost of $5.7M. This total cost includes approximately $225K in salary (and relevant overhead such as employer taxes and benefits), the remainder includes standard overhead margin to pay for operational expenses like legal, HR, finance and other support as well as some profit margin to the companies, to ensure long term sustainable growth. Overall, this corresponds to an hourly rate of around $165, a mid-market rate for consultancies.

We will include details on the number of employees working on the Hub in our quarterly reports. In case we fall below the allocated number of employees during a quarter (due to turnover etc.), we will only withdraw the funding from the multisig needed for the number of employees actually employed.


Schedule of work

Like Jehan says, several of the items in our roadmap are closely integrated. I’ll speak to the maintenance (both Gaia and ICS) and testing components since those are the ones that have a clear schedule.

This is a quarterly schedule for two releases per quarter, equating to a total of eight releases over 2024. At this point, we can’t predict or commit to exact features or core tech version updates for each release – we don’t have the information necessary to make those decisions or estimates yet.

Similarly for ICS support at Hypha, we can’t accurately predict how many chains we might be onboarding (however, we expect this to be in the range of 2-5). For a sovereign-to-consumer transition rehearsal, this is what our schedule looks like:

This work occurs for every rehearsal and must fit into the continuous Gaia release cycle – launch rehearsals, Gaia upgrades, and mainnet upgrades all happen on Wednesdays in approximately the same 15:00 UTC time slot to reduce coordination overhead. We have approximately a two week process for a launch rehearsal.

As part of ICS maintenance and testing, Hypha onboards incoming consumer chains to the Hub’s Replicated Security testnet. Each launch rehearsal has a runway of at least two weeks during which Hypha is on-call and supporting consumer chains both in putting together their Replicated Security chain parameters and approaching the Hub governance process. Hypha works most closely with consumer chain teams during the onboarding process, and then again at mainnet launch when we are on-call alongside the Informal Hub team to troubleshoot, debug, and support their launch on the Cosmos Hub.

Testnet Wednesday events include Gaia upgrades, consumer chain upgrades, consumer chain launch rehearsals, and consumer chain sovereign-to-consumer transition rehearsals. During sovereign-to-consumer rehearsals, Hypha works closely with the consumer chain team and testnet validators over the course of two hours for a coordinated launch.

Our testnet work is measurable less with a deliverable and more with the operational smoothness experienced by validators and the incoming consumer chains onboarding onto the Hub. However, we do publish our Testnet Wednesday reports here on the Hub forum.


Testing and maintenance have the overall goal of steady, consistent progress in the development of the Hub. We propose these as metrics for measuring against that goal:

  • Smooth upgrades measured in time per upgrade
  • Testnet operational smoothness measured in time taken every Testnet Wednesday
  • Testnet participation measured in validator participation during upgrades and consumer chain launches
  • Quality of learnings measured in testnet reports


Hypha’s current headcount is all focused on the testing and testnet line item. If we work strictly on testing, our 2024 headcount would be 2 - 3 FTE. Over 2024, we plan to be more involved with Gaia maintenance and release integration work, which would push our headcount to the upper end of our estimation (3 - 4) as we take on tasks such as release integration.

On Hypha’s side, all the FTE funded by this proposal are working directly towards our KPIs as eng/ops. We aren’t funding separate marketing or BD out of this budget, though the rate does include general operational expenses.

Ongoing funding and oversight structure

Like Spaydh says, the overall format of core development funding on the Hub is probably going to take many iterations of experimentation. Between RMIT’s recent essays on ICS governance and Noam’s DRIP proposal, there are lots of really interesting directions to investigate.

This is one of the reasons that our proposal is only for a yearlong funding period – by the time we are making a second proposal, research in this area will likely have progressed and we’ll have more options.

Direct responses

Can you please explain how Informals proposal and Notionals proposal differ? Or is this a ‘double spend’ for this area?

- Rarma

Afaik (and Lit or Jacob can chime in here to correct me if need be), Notional’s scoped work is:

  • Monitoring the Hub for vulnerabilities
  • Coordinating with Hub teams to produce a security patch and assist in rollout
  • Advising core teams of issues in upcoming releases for safety or liveness
  • Improvement and development of products like the Hub

Monitoring the Hub is something many teams do, and it’s good for many teams to do this. There’s warranted overlap here. Similarly, improvement and development of products like the Hub has some overlap - it’s an open source repo and many people contribute to it.

However, the other items involve coordinating or advising the core Hub team (that’s us right now). We work with the Notional team on identifying and resolving issues that affect the Hub. For example, Notional has recently been doing security testing using the testnets operated by Hypha and we have been closely in contact throughout this process and potential resolutions for it on the Hub.