Changelog
- 2023-Sep-27: Posted initial draft
- 2023-Oct-13: Adjusted the naming of opt-in security research topics
- 2023-Oct-18:
- Indicated ‘go to market’ status for IBC Routing and Mesh Security as part of the R&D budget.
- Incorporated further details about headcount in the Budget section
- 2023-Oct-20:
- Added performance metrics
- $5.7M budget will be held in 30% ATOM, 70% USDC
- No bonus will be paid out unless teams exceed expectations for the quarter
- Any bonus earned will be placed into a vesting account with a one year lock
- Added specificity to Hypha’s headcount
- 2023-Oct-20: Flipped to
last-callwith intent to go on-chain on October 26. Fixed internal links. - 2023-Oct-25:
- Defined our 3rd party module maintenance responsibilities.
- Defined our marketing and communications work for the Hub.
- 2023-Oct-26:
- Added ATOM numbers to prepare for going on-chain
- Corrected some mentions of ‘opt-in security’ to ‘partial set security’
- Added detailed liquidation steps
- Added Jim Parillo to the oversight committee, replacing ‘representative from Imperator’
- Updated performance metrics for Interchain Security
- Added oversight committee names and liquidation multisig names to on-chain summary text
Table of contents
- Summary
- Background
- Roadmap
- Accountability
- Budget
- Appendix A: Past work
- Appendix B: Quarterly reporting format
- Appendix C: Grading
Summary
This is a proposal to fund a core development and testing package on the Cosmos Hub for 2024. The teams that would receive funding if this proposal passes are the Cosmos Hub Teams from Informal Systems and Hypha Worker Co-operative. The total budget is $5.7 million USD (to be held in 30% ATOM, 70% USDC), plus 100k ATOM in performance bonuses, for a total of 914,285.71 using a spot price of 7 USD. This funding would replace both teams’ current funding from the Interchain Foundation and have these teams be directly accountable to the community. As of October 26, 2023, this represents approximately 18% of the community pool.
Notes:
- ATOM values have been calculated using a spot price of $7 USD as of October 26, 2023.
- This budget only covers the teams at Hypha and Informal that work specifically on the Cosmos Hub.
- The ATOM total requested from the community pool includes a 25% buffer on the budget amount to cover fluctuation in ATOM price over the 2-week voting period. Any unused buffer will be returned to the community pool.
This proposal is summarized by the following propositions:
- The release of 1,117,857.14 ATOM to finance:
- An overall budget of $5.7M USD plus 100k ATOM in possible performance bonuses, with a 25% buffer to cover price fluctuations between start of voting and liquidation to USDC. Any unused buffer will be returned to the community pool.
- Maintenance and development of the Interchain Security protocol
- Maintenance of the Cosmos Hub Gaia core software
- Maintenance and improvements to the testnet program for the Cosmos Hub and its consumer chains
- Research and development for improvements to Interchain Security such as Partial Set Security, Mesh Security (go to market), Atomic IBC, and IBC routing (go to market)
- Thes ATOM will be released to a liquidation multisig responsible for converting the appropriate portion of ATOM into USDC via the process laid out in the full text of the proposal. The liquidation multisig is made of:
- Simply Staking
- Stakecito
- Citadel One
- CryptoCrew
- Stakin
- The ratification of a committee that will oversee the work of Informal and Hypha within the time period of January 1, 2024 - December 31, 2024 and provide insight and visibility to the community about that work. The committee is made of:
- Stride contributor: Aidan Salzmann
- Neutron contributor: Avril Dutheil
- Polkachu representative: Polkachu
- Jim Parillo at Figment Capital
- Shane Vitarana at Stargaze
- James Hinck, Product Manager at Circle
- An optimistic vesting mechanism which gives the community the power to suspend either team’s work agreement at any time via a governance proposal if that team is not fulfilling their commitment to the Hub
Details of disbursal of funds:
- The split between Informal and Hypha is as follows:
- Informal budget: 3.16M USDC + $1.34M worth of ATOM
- Informal bonus: 79k ATOM
- Hypha budget: 840k USDC + $340k worth of ATOM
- Hypha bonus: 21k ATOM
- Of the budget portions, any unused funds will be returned on a quarterly basis.
- Any unearned bonuses will be returned on a quarterly basis. Earned bonuses will go into a one year locked vesting account.
Governance votes and outcomes
The following items summarize the voting options and what it means for this proposal:
YES
You agree to fund the Informal Systems Hub team and Hypha Worker Co-operative with $5.7 million USD + 100k ATOM from the Community Pool for the Cosmos Hub’s continued development, maintenance and testing over the period of January 1, 2024 - December 31, 2024, held accountable by their specified oversight committee.
A ‘YES’ outcome will immediately release 1,117,857.14 ATOM to the specified wallet.
NO
You do not agree to fund the Informal Systems Hub team and Hypha Worker Co-operative based on the terms of this proposal.
A ‘NO’ or ‘NO WITH VETO’ outcome will not fund the Informal Systems Hub team and Hypha Worker Co-operative from the Cosmos Hub community pool.
NO WITH VETO
A ‘NoWithVeto’ vote indicates a proposal either (1) is deemed to be spam, i.e., irrelevant to Cosmos Hub, (2) disproportionately infringes on minority interests, or (3) violates or encourages violation of the rules of engagement as currently set out by Cosmos Hub governance. If the number of ‘NoWithVeto’ votes is greater than a third of total votes, the proposal is rejected and the deposits are burned.
ABSTAIN
You wish to contribute to the quorum but you formally decline to vote either for or against the proposal.
Background
The Hub’s Past and Present: a multi-stakeholder effort
Both Informal Systems and Hypha Worker Co-operative have played critical roles in the go-to-market strategy of Replicated Security, which is the core product offering of the Cosmos Hub.
The Cosmos Hub team at Informal Systems played a key role in the design and development of Interchain Security, and as of January 2023 also took on many duties related to maintaining the Cosmos Hub software and preparing new releases. Hypha has supplemented this work via thorough testing for all releases, operating the Replicated Security testnets program, and providing critical support to incoming consumer chains. Here are some of the teams’ past accomplishments:
- Designed, developed, and tested the Replicated Security protocol
- Maintained a steady cadence of smooth Cosmos Hub releases
- Did business development to onboard a top tier set of consumer chains
- Did cutting-edge research on shared security and liquid staking technology and economics
- Developed innovative new simulation and testing techniques to ensure excellent security and uptime
See a more detailed list in appendix A.
Together, these independently operating organizations touch every step in the software development and release process on the Cosmos Hub, which has resulted in the on-boarding of top-tier consumer chains such as Neutron and Stride, with many more waiting to join.
Software released by Informal Systems and tested by Hypha has enabled the Cosmos Hub to maintain a regular cadence of upgrades and stay up to date with the latest versions of its dependencies. Informal and Hypha have also led the emergency release process, smoothly bringing critical upgrades to the validator set when necessary.
The budget in this proposal will allow the Hub Teams at Informal Systems & Hypha to continue their work maintaining the Cosmos Hub, improving Interchain Security, on-boarding consumer chains, and spearheading the development and testing of new features.
Becoming accountable directly to the Cosmos Hub community
During 2023 and before, development of the Cosmos Hub was funded primarily by the ICF. The Informal Hub Team’s budget in 2023 was around $4.5 million, while Hypha’s was approximately $1.2 million.
Funding development of the Cosmos Hub from its own community pool would mark a milestone of decentralization and sustainability, and would allow the Hub development team to be directly accountable to the Cosmos Hub, rather than to the ICF. Maintaining independence from the ICF enables the Hub to set its own standards for innovation, execution, and accountability.
With four years of governance experience, there is no entity better equipped to be directly responsible for funding the development of the Cosmos Hub than the Cosmos Hub community itself.
Roadmap
There are many exciting developments for the Cosmos Hub in progress, but the Informal and Hypha teams have a very clear focus. We work on the lowest level of the stack on the shared security, atomic composability, and routing technologies which underpin the Atom Economic Zone. Our roadmap for 2024 focuses on:
- Continuing to maintain the Cosmos Hub, cutting releases, working with teams to release new features, etc.
- Continuing to perfect Interchain Security to make the protocol more robust, sustainable, and scalable for validators to operate.
- Creating procedures and software to ensure reliable and well tested Cosmos Hub and consumer chain releases, by using simulations and several different kinds of testnets.
- Research and development to expand the market for existing features and bring new features to the Cosmos Hub
- Research Partial Set Security to reduce costs on the validator set, and allow a greater number of consumer chains to join.
- Research Atomic IBC to bring lower cost operation and atomic composability to Cosmos Hub consumer chains.
- Assistance on Mesh Security development, deployment on the Hub, go to market, and research on complementary features to give the Hub a new way to share its security and even consume security.
- Go to market and deployment for ICS33 IBC routing to allow Cosmos chains to route IBC packets transparently over the Hub to reduce IBC costs throughout the ecosystem.
Steward the Cosmos Hub
The Informal and Hypha Hub Teams will continue to contribute to stewardship of the Cosmos Hub’s core software, Gaia. This is obviously of core importance to everything else on this roadmap, and the team will continue to uphold the high standards set in 2023.
Enable collaboration for the Hub across the ecosystem
Many different features for the Cosmos Hub are being planned and developed by teams around the ecosystem. The Informal Hub Team has played a key role over the past year in advising, reviewing, and integrating features developed by other teams. Taking the Liquid Staking Module as an example:
- Analysis: We did extensive analysis and research on the possible risks of liquid staking.
- Product: We then advised the Stride and Iqlusion teams on the best ways to reduce these risks, informing the addition of features such as the global cap on liquid staking.
- Organizational: We worked with the Cosmos-SDK, Stride, and Iqlusion teams to decide which repository the LSM code would be housed in, and who would maintain it.
- Code review: We helped review the LSM code before its deployment on the Cosmos Hub.
We will continue providing this level of support and review for other new features.
Regular release schedule
Maintaining a regular release schedule for the Cosmos Hub has been core to our approach in the past year. Regular releases reduce the risk of any given release, build “muscle memory” for releases in the team and the validator set, and ensure that features are deployed on a timely and regular basis.
Validator support
Informal and Hypha are on-call during both planned and emergency Hub upgrade deployments. We communicate with validators about the software, as well as troubleshoot any issues that come up. Due to this proactive management, Hub upgrades have been very smooth, and have resulted in very little downtime. This was even the case for the one emergency upgrade we have had this year, which resulted in only around 5 minutes of downtime. The smoothness of the actual upgrades is the result of weeks of preparation by our team and proactive communication with a large number of validator teams.
Review dependencies and update the Hub
We are keeping the Cosmos Hub up to date with the latest versions of Cosmos-SDK, IBC, and other dependencies. This also includes deciding when a dependency is ready for deployment, given the Hub’s high standard of security. For example, we updated the Hub to SDK 47, as well as spearheaded the effort to get a 3rd party audit on critical parts of the update.
Marketing and communications
Marketing activities include and are limited to communications around the work and development on the Hub done by Hypha and Informal Systems within the scope of this proposal. Some of these items include Community Developer Calls, episodes of Informal Spaces, Monthly Updates and all of the supporting marketing activities around each of these initiatives. Informal Systems marketing will assist in the marketing of the Hub itself and the expansion of business activities within the scope of the collaborative initiative laid out in this tweet.
Interchain Security
We will continue to perfect and maintain ICS, which is the cornerstone of the AEZ. Our improvements will focus on increasing reliability and performance of the protocol
General maintenance
ICS must work with several versions of IBC and Cosmos-SDK, since different consumer chains use different versions of these libraries. Maintaining releases compatible with these versions and porting features from one to another is not a small task but is critical to making sure that all consumer chains run smoothly.
Epochs
The consumer chain validator set is currently updated every block. This is best for reasoning about security, but it generates a lot of traffic. We’d like to reduce this to a much lower frequency, for example once an hour. We have to prove that this is secure, but once this is done it will provide a more efficient implementation of ICS.
Read-only protocol
We would like to simplify the ICS protocol to work with IBC queries. This will allow for a much simpler protocol, and easier maintenance going forward, as well as making the Hub more robust to malfunctioning consumer chains. The read-only protocol will also make it much easier to implement features such as opt-in security, and to move faster in general.
Documentation
We maintain the documentation at https://cosmos.github.io/interchain-security/, which provides a technical and theoretical overview, as well as a guide for validators and consumer chain teams.
Testing
The cultures of Informal and Hypha are rooted in software verification and QA, and we put a lot of emphasis on testing in our work on the Cosmos Hub. Beyond our extensive test suites for Interchain Security, we also write some of our own tooling.
CometMock
CometMock is a stand-in for Comet that can be used during testing which greatly shortens test runtime. Many Cosmos projects have end to end tests that run for 20-30 minutes, and CometMock can reduce this to 1-2 minutes. CometMock also provides determinism (tests run the same exact way every single time), and complete control over time and block production. We built CometMock for use on the Hub, but we are working to bring it to a broader audience of Cosmos projects.
Simulating Cosmos Hub state for testing
Hypha has developed a testing framework to export and modify the Cosmos Hub state. We use this framework to conduct rigorous tests on specific Gaia branches, simulating upcoming mainnet upgrades as closely as possible. This approach significantly reduces the risk associated with new releases, ensuring a high level of confidence when launching them on the Hub and contributing to the overall safety and security of launches. We will continue to enhance this framework, extending its coverage to critical scenarios and using it to test upcoming Gaia releases.
Three-phased testnet release process
Every Cosmos Hub release undergoes a thorough testing process across three phases of testnets. In the development phase, release testnets are automatically deployed using Hypha’s continuous integration scripts. Upon completion of the first release candidate, we conduct local testnets and execute a predefined set of tests. Finally, we upgrade two public testnets with Cosmos Hub validators, enabling extensive community testing. The public replicated security testnet supports multiple consumer chain testnets, facilitating integration testing in a production-like environment. Hypha will continue to oversee the Cosmos Hub and consumer chain releases through this testnet process while maintaining the necessary infrastructure and tools for public testnets.
Testnet Wednesdays for increased validator engagement
Hypha has matured Cosmos Hub’s testnet program, conducting over 50 testnet events in the current year. These events serve as valuable training grounds for validators, software testing, and the identification of operational challenges related to the Gaia codebase. In 2023, we introduced “Testnet Wednesdays,” an initiative aimed at regularizing public testnet events. This initiative allows validators to allocate dedicated time for participation, thereby boosting validator engagement. In 2024, our plan is to further refine Testnet Wednesdays by diversifying event types to cover various testing aspects such as performance, security, and functionality.
Chaos testing for scalability and resilience
As part of routine testing on both local and participatory testnets, we intend to introduce chaos testing patterns. This approach will help us explore operational limits, discover scale-related vulnerabilities, and identify system bottlenecks.
Research and development
The items in this section of the roadmap are still under active research and are more likely to be substantially modified, added to, or deprioritized. Over the course of 2024, we will focus on one or two of these topics to begin building and putting into production.
Opt-in Security
Opt-in security is a family of techniques which could allow fewer validators than the whole set to validate a given consumer chain. This has potential cost and performance benefits, increasing the flexibility of the protocol, while still being a more “full-service” offering than Mesh Security. We will research techniques with which this can be accomplished, and use them to design a protocol that can benefit new and existing consumer chains. There are two basic techniques we are looking at to accomplish this:
Opt-In with Fraud Proofs
In the case where only the stake of the opted-in validators is used to secure the consumer chain, we encounter the subset problem (discussed on the Informal blog here). To be secure it needs something to prevent incorrect execution. This can be done by fraud proofs, validity proofs, or fraud votes. This requirement also exists for mesh security.
Partial Set Security
This research idea has entered the CHIPs discussion phase on the forum. Validators who are not opted-in need to delegate their stake to validators who are opted-in. This allows the full stake of the Hub to secure consumer chains, as well eliminating the need for fraud proofs, validity proofs, or fraud votes. This idea has previously been discussed under names like ‘partial opt-in’, ‘proxy opt-in’, or ‘subset security’.
Atomic IBC
Atomic IBC is a recently released plan that sets the Cosmos Hub up for long-term success. It does this by combining the scalability and sovereignty of the multichain ecosystem with the atomic composability of a smart contract platform like Ethereum. This combination of scalability and composability has long been a goal of many blockchain designers, and Atomic IBC solves it elegantly.
Atomic IBC will give participating consumer chains the ability to integrate seamlessly by composing multi-chain IBC interactions with many steps and delays into one transaction, called an “atomic bundle” which executes across many chains in one step. This can reduce the code required to implement complicated multi-chain workflows like those used by Timewave by around 90%.
One way to make Atomic IBC happen is by having consumer chains share a blockchain with bigger blocks but parallel execution of most transactions. We’re calling this the “Megablocks” architecture. The Megablocks architecture is relatively simple, will provide the best user experience possible, and will reduce the cost of running consumer chains using this architecture.
However, Megablocks limits the amount of customizations that consumer chain teams can make to their low-level Comet configuration and code, and necessitates a shared mempool. For this reason, we are also researching other techniques to enable Atomic IBC, from synchronizing block production across several Comet processes, to heterogeneous Paxos, a consensus technique which lets chains make shared blocks containing atomic transactions when required.
Consensus research
We will work towards building a proof of concept of the Megablocks architecture, while also exploring other techniques such as block synchronization and heterogenous Paxos to enable greater consensus protocol customization. We will also work to rigorously define how Atomic IBC will interact with the low level consensus customizations offered by ABCI++, and make sure that UX benefits offered by Atomic IBC are worth it.
Developer and user experience research
We will explore what the developer interface to Atomic IBC could look like, and how it could make existing and future multi-chain composition use cases easier and better. We will work with existing and prospective consumer chains to see how these UX improvements can benefit their applications.
Protocol design
We will work on specifying and building the Atomic IBC protocol. This will entail changes to IBC to allow it to pass messages sequentially between Atomic IBC consumer chains as part of an atomic bundle, and calculate gas costs for atomic bundles. This will also entail changes to Comet to allow it to roll back transactions in an atomic bundle if any of the other transactions error.
Mesh Security (go to market)
The Informal Hub Team has lent our expertise to the Mesh Security team, on topics such as slashing and fraud proofs. We will continue to collaborate with them to further the development and launch of Mesh Security and share concepts between the protocols to ensure that the Cosmos Hub can be well positioned for a mesh-security future.
IBC routing (go to market)
One of the Cosmos Hub’s original missions is to route IBC packets between other Cosmos chains to increase the efficiency of the interchain by reducing relaying and light client update costs. This has been delayed by the lack of working routing in IBC. Recently, a workable IBC routing scheme has been proposed by the team at Polymer. The scheme allows IBC packets to be routed over the Hub without requiring any data to be written to the Hub, by aggregating IBC light client proofs from many connected chains.
This will augment the packet forward middleware already installed on the Hub by providing transparent routing – applications will have to make no distinction between direct and routed packets. For example, tokens transferred with this style of routing will be indistinguishable from, and fully fungible with direct routed tokens.
The Informal Hub Team will lead an effort to get this deployed to the Cosmos Hub, the Hub’s ICS consumer chains (including those using Atomic IBC), and offer it to other chains in the Cosmos ecosystem.
We will consult with the Polymer and IBC-Go teams to find the best path to deploying this feature. We will also help implement any additional code that is required. We will support existing ICS consumer chains in deploying this feature to reduce their relaying costs. We will also work to educate the broader community about it and introduce the feature to non-consumer chains.
Accountability
Performance metrics
Research and development
Goal: Bring innovative features to the Hub
Suggested metrics:
- Proposals progressing through CHIPs process
- New features included in Gaia releases
Interchain Security
Goal: Make the Hub an appealing and sustainable provider chain
Suggested metrics:
- Cost reductions for operators measured in number of consumer chains
- The Hub being an attractive provider chain, also measured in number of consumer chains
- Quality metrics of consumer chains (e.g., TVL, market cap, social media sentiment)
Gaia maintenance
Goal: Keep the Hub up-to-date and running smoothly
Suggested metrics:
- Consistent releases over the funded year (est. 2 releases per quarter)
- Inclusion of core stack version updates in Gaia releases, pending stack version releases (est. 2 sdk updates, 2 ibc updates)
- Time per upgrade
Testing and testnets
Goal: Ship bug-free code to mainnet and improve operational efficiency
Suggested metrics:
- Security and liveness issues addressed using the testnets
- Consistent testnet events (upgrades, launches, wargaming sessions)
- Testnet participation measured in validator participation
- Quality of learnings measured in testnet reports 1
Oversight committee
This committee exists to oversee, advise, and guide the core teams and the community in regards to the Informal Systems and Hypha Co-op Hub teams’ work throughout their fulfillment of their Cosmos Hub development, maintenance, and operational work related to this proposal.
This committee is ratified by the passage of this funding proposal. The committee represents various stakeholder groups in the community that are equipped to perform oversight to these teams, are impacted by their work, or able to provide an outside perspective. It includes representatives from consumer chain teams (representing the Hub’s “customer” voice), Hub validators (representing the Hub’s “operator” voice), and prominent projects in the industry who can provide an external point of view.
The committee’s power ultimately comes from the confidence and support of ATOM token holders. If, at any time, the token holders believe that the committee is not providing sufficient oversight or that either team is not fulfilling their commitment to developing the Hub, a governance proposal can be passed to dissolve the committee, suspend funding for either or both teams, and return unused funds to the Community Pool.
With these things in mind, the committee will:
- provide oversight to the Hub teams at Informal Systems and Hypha Worker Co-operative in their capacity as core teams of the Cosmos Hub for the funding period of January 2024 through December 2024.
- assess work done by the Hub teams at Informal Systems and Hypha Worker Co-operative. This assessment will be based on reports written by Hypha and Informal (see appendix B) and will result in grades submitted after each quarterly meeting (see appendix C). These reports and grades will be made available to the community.
The committee will not:
- act as a body overseeing any Hub development aside from that of Informal and Hypha. Other community pool funded efforts are out of scope, and may have their own committees or other governance structures.
- act as a lasting governing body. The committee is effective for the duration of the proposal (from January 1, 2024 - December 31, 2024). The committee, together with the core teams, are put in place by governance approving this proposal.
- make final decisions on strategic direction or roadmap. These decisions should be guided by Informal and Hypha and ultimately approved by the Hub’s community and customers.
Responsibilities and Output
The oversight committee will convene for 5 meetings during the 2024 year, with one meeting at the beginning of the year, and one meeting at the end of each quarter. Each meeting will consist of presentations by representatives of Informal and Hypha, followed by questions and discussion from committee members. Committee members and core team members will communicate asynchronously to iterate on feedback given during these quarterly meetings.
Meeting minutes and summaries will be produced by the core teams and made available to the community after being approved by the committee along with the teams’ quarterly reports, and the committee’s grades on the quarter’s progress. The format of the teams’ reports and the grading system are shown in appendix B and appendix C.
2024 roadmap meeting
Occurs at the beginning of the year (January 2024) to discuss and formalize roadmaps for each team.
- Presentation on 2024 product agenda
- Presentation on 2024 operational and testing goals
- Questions and discussion from committee members
Quarterly assessment meetings
Meeting at the end of each quarter (i.e., Mar 2024, Jun 2024, Sep 2024, Dec 2024) to assess road map progress and undertake the review & reporting process.
- Presentation from Hypha and Informal on the quarterly report (see appendix B: Quarterly reporting format)
- Questions and discussion from committee members
- Within a week: Submission of grades by committee members, signoff by committee on meeting summary, and posting of quarterly report, meeting summary and grades on forum.
Membership
The membership of the committee will not change throughout the funding period named in this proposal. The members of the committee are as follows:
- Stride contributor: Aidan Salzmann
- Neutron contributor: Avril Dutheil
- Polkachu representative: Polkachu
- Jim Parillo at Figment Capital
- Shane Vitarana at Stargaze
- James Hinck, Product Manager at Circle
Procedures around team removal
A core part of the committee and the community’s oversight over the team rests on their ability to remove the team and end funding. However, this should take place in a smooth framework so that the team has the ability to correct performance issues if any exist, before the “nuclear option” is invoked.
If, for two consecutive quarters, the committee submits a grade of “Needs improvement” in the overall performance category, they must also write and submit a signaling proposal calling for removal of the team and suspension of the funding.
Hypha and Informal submit reports separately, are graded separately, and can be removed separately.
As a reminder, any member of the community can submit a proposal to remove funding at any time, without following this procedure.
Budget
Our total budget for 2024 is $5.7 million, which will be held in 30% ATOM, 70% USDC. This budget covers the following broad areas listed below, and this is roughly how we expect to distribute effort within these areas. While these line items are not exact, they represent the approximate time investment from our teams on each of these areas.
| Budget | Notes | |
| R&D | $1,700,000.00 | Partial Set Security, Atomic IBC, Mesh Security, IBC Routing |
| Interchain Security | $1,500,000.00 | Read-only protocol, epochs, general maintenance and support |
| Gaia maintenance | $1,500,000.00 | Upgrades, release management & incident response |
| Testing and testnets | $1,000,000.00 | Simulated testing, Cometmock, testnets validator outreach |
| Total | $5,700,000.00 |
This request aligns closely with the 2023 funding allocated by the ICF, maintaining a consistent team size. The rates charged are in line with standard market rates and encompass various costs such as salaries, operational expenses, and team incentives in ATOM.
This budget is crucial to support the growth and competitiveness of the Cosmos Hub, while ensuring ongoing motivation and dedication from our team members. The total Hub team’s headcount will typically range from 14 to 20 individuals (2-4 for Hypha, 12-16 for Informal) with varying levels of experience, subject to occasional fluctuations due to turnover and hiring cycles. On a quarterly basis, both Hypha and Informal will provide reports outlining the amount of funds used to date for team members directly involved in the project (see appendix B). Rates are aligned with the mid-point of mid-market software development rates which typically range from $110-$220/hour.
Headcount
The Informal Hub Team budgets for 14 team members. This includes:
- A project lead, product owner, and operations manager, who oversee and participate in every part of the team’s work.
- 5 generalist software engineers, who write most of the code.
- A protocol designer, who answers complicated questions about protocol correctness
- A verification engineer who builds and uses automated testing technology like CometMock, Quint, and Model-based testing
- A finance/economics researcher who answers quantitative questions about the financial impacts of technologies (like liquid staking)
- A devops engineer who helps Hypha with production operations and works on performance related aspects of the code and infrastructure
- 1 FTE equivalent for marketing and BD support, to help communicate about the Hub and our work, and find and guide prospective consumer chains.
The Hypha Hub Team’s budget includes:
- 2 general software engineers who write code, design and build tests, and work with other dev teams
- 1 product manager who manages scope, timeline, communications, and is involved with every part of our team’s work
- 0.5 devops engineer who manages and orchestrates our fleet of nodes on two public testnets and multiple consumer chains
Third party module maintenance
For Hypha, moving into Gaia maintenance and release integration would involve increasing our headcount to 4. Hypha plans to allocate 1 FTE towards Gaia maintenance. Of this, 0.5 time will be dedicated to release integration and cutting release candidates. Remaining 0.5 time will be dedicated to maintenance of governance approved Cosmos Hub modules developed by third party developers (not by Informal or Hypha) which must be integrated into the Gaia binary. We encourage developers to submit feature ideas and code for integration into Gaia, preferably through the CHIPs process. Although our recommendation is that third-party developers maintain their modules, we have budgeted dedicated time to support their maintenance. This allocated time will ensure we can perform routine upkeep and manage module dependencies promptly. However, if the maintenance requirements of third-party modules exceed our budget, it may lead to release delays or a deprioritization of other tasks as we reassign developers.
Per-head rate
For both organizations, our per-head total rate averages around $325K per year, this is how with Informal’s 14 and Hypha’s 3.5 team members, we get to a cost of $5.7M. This total cost includes approximately $225K in salary (and relevant overhead such as employer taxes and benefits), the remainder includes standard overhead margin to pay for operational expenses like legal, HR, finance and other support as well as some profit margin to the companies, to ensure long term sustainable growth.
Reporting
We will include details on the number of employees working on the Hub in our quarterly reports. In case we fall below the allocated number of employees during a quarter (due to turnover etc.), we will only use funding from the multisig needed for the number of employees actually employed.
Performance bonuses
The 100k ATOM performance bonus will be awarded quarterly (up to 25k per quarter) based on the “Overall performance” grading category (see appendix C), which is determined based on several factors including KPIs set in the previous quarter.
- “Needs improvement”: 0% of bonus awarded
- “Meets expectations”: 0% of bonus awarded
- “Exceeds expectations”: 100% of bonus awarded
If a quarterly performance bonus is earned, it will be withdrawn from the yearlong vesting accounts by each team and immediately transferred to a vesting account with a one year lock on it to further incentivize good performance from the teams. If the performance bonus is not earned, it will be immediately returned to the community pool.
Funding Control Mechanism
Implementation using classic Cosmos-SDK vesting multisig
This implementation involves several compromises because of the inflexibility of the classic SDK 0.45 vesting multisig currently running on the Cosmos Hub.
- Vesting schedule: To approximate payments at the beginning of the month, vesting will start one month early, in December 2023. This means that the full amount necessary to meet payroll for the month will be available to the core teams by the beginning of January 2024, and every month thereafter, with full vesting complete at the beginning of December 2024.
- Multisig control: Each funded team will control their own multisig vesting wallet. Core team representatives will only be able to withdraw funds that have already vested. Withdrawals occur on a monthly basis.
- Dissolution: To end funding for a particular team, an upgrade proposal containing migration code moving the funds out of the vesting multisig and back to the community pool must be passed via governance on the Cosmos Hub. It is possible for anyone to introduce this proposal, but doing so is an explicit part of the mandate of the committee.
Liquidation (step by step process)
- If/when proposal passes: Total ATOM amount will be sent from community pool to a 3 of 5 interim multisig comprising:
- CryptoCrew
- SimplyStaking
- Stakecito
- Stakin
- Citadel One
- The interim multisig will use an OTC deal or swap to liquidate $4M worth of ATOM (using the 25% buffer to cover price fluctuation and fees if needed) into USDC. This $4M represents 70% of the total budget. The remaining 30% will be retained in ATOM. The end result of this step is that 5.7M USD worth of tokens (70% USDC, 30% ATOM) will be made available to send to the teams.
- The interim multisig will send the following transactions to vesting accounts managed by the respective teams. While all accounts are continuously vesting over 2024, the ATOM budget and ATOM portions are divided for ease of transparency and reporting.
- Informal budget: 3.16M USDC + $1.34M worth of ATOM
- Informal bonus: 79k ATOM
- Hypha budget: 840k USDC + $340k worth of ATOM
- Hypha bonus: 21k ATOM
- The interim multisig will return all remaining ATOM buffer to the community pool.
- If the ATOM price is exactly the same as when the prop went up for voting (2 weeks before this point), then the entire 25% buffer will be returned to the community pool.
- If the ATOM price went down less than 25% over the 2 week voting period, then some of the buffer will have been used to cover the difference, and less will be returned.
- If the ATOM price went down more than 25% over the 2 week voting period, the scope of work and/or headcount may need to be reduced based on the available budget. We will work with the community and our oversight committee to get feedback on this.
- If the ATOM price went up over the 2 week voting period, then more than the 25% buffer will be returned to the community pool – the interim multisig will only sell enough ATOM to cover the amount specified in the funding prop.
- The interim multisig will publish a full description of the deal, including documentation of:
- Trading venue for the OTC deal
- Price received in the OTC deal (it may be necessary to offer a slight discount from ATOM’s spot price)
- Any trading or other fees involved
Appendix A: Past work
Informal Systems came up with the original protocol design and specification for Interchain Security in 2021, and led the implementation of Replicated Security shortly thereafter. Replicated Security is a complex piece of software which includes the core provider and consumer modules, a protocol to allow consumer chains to have their own unbonding periods, key assignment, and the sovereign-to-consumer migration feature which allows a transition to Replicated Security without disrupting IBC connections.
Interchain Security has become the Cosmos Hub’s flagship product and we continue to lead its development through six releases of Gaia (v8 through v12) this year. Informal has also published research on Shared Security and comparisons of different security options for the Hub.
Together, Hypha and Informal have implemented several testing frameworks for Replicated Security, making QA and verification much easier for builders using this technology.
In 2022, Hypha re-established the Cosmos Hub Testnet program and has since built it into a thriving testing ground for every consumer chain on the Hub. We coordinated Cosmos Hub’s third incentivized testnet, _Game of Chain_s, and continued that work into the Replicated Security Testnet. This testnet has been the home of ten launch rehearsals, four synchronized upgrades, and one re-launch; it is an instrumental part of the onboarding process for Hub consumer chains.
Hypha has continued to publish educational content and analysis of Replicated Security (Swiss Booklet: Neutron, Preparing for Replicated Security, Conditional Basic Income) throughout 2023.
For a detailed breakdown of our teams’ past work, see here this document.
Appendix B: Quarterly reporting format
Click to open Appendix B: Quarterly reporting format
Hypha and Informal will submit and present separate quarterly reports, and will be graded separately.
Work done in previous quarter
This is an overview of work done by the team in the previous quarter, and a review of how it matched up to the plan for the quarter.
Highlights
Completed tasks and successes.
Challenges
Tasks that took longer than expected, or turned out to be more complicated. Unexpected events such as security incidents, etc.
Adjustments
Places where we deviated from the quarter’s plan, due to better understanding of tasks involved, unforeseen circumstances, and strategic adjustments.
Plan for next quarter
Taking into account the work and events of the previous quarter, what is planned for the next quarter? This is also guided by the overall roadmap and long term strategic goals from the roadmap.
Major areas of work
What are the major focuses this quarter? What are some individual tasks or deliverables that will be worked on? What KPIs will this work be judged on?
Expected challenges
Some tasks involve a higher level of uncertainty than others. This is a place to highlight expected complications in the work.
Community input
One of our core roles is to build what the Cosmos Hub community wants. We also take the role of leading through research and development, but ultimately, everything deployed on the Hub is subject to community approval.
Input received
In which ways has community feedback been solicited in the past quarter, and on what? What is the feedback?
Interpretation and incorporation of input
How has this feedback modified the work executed this quarter, next quarter’s plan, and the year’s roadmap?
Operational report
Baseline maintenance and regular processes are a significant part of the teams’ work. Operational tasks are not necessarily dependent on the roadmap or subject to shifting over quarters, but reflect the consistency and reliability of core operations.
Upgrades
What Hub upgrades, emergency or planned, have happened? How did they go?
Consumers
What consumer chain testnets, launches, and upgrades have happened?
Security events
Were there any security events that happened? What was the outcome and how were they fixed?
Funding report
Burn rate
How much was spent this quarter? (of course, the vesting account puts an upper limit on this)
Remaining funds
How much remains for the year?
Appendix C: Grading
Click to open Appendix C: Grading
The committee will assess the work of the previous quarter in four categories, and give a grade for each one, along with an overall grade. There are three grades:
- Exceeds expectations - the team is going above and beyond their commitments
- Meets expectations - the team is performing their job and fulfilling their commitments to the Hub
- Needs improvement - the team is falling short of their commitments
These are the four categories, which roughly correspond to the areas covered by each report and result in an overall performance grade:
- Meeting last quarter’s plan
How well did the team execute against the plan and KPIs set at the last meeting?
- Alignment and progress on overall roadmap
How well do last quarter’s execution and this quarter’s plan fulfill the high level roadmap?
- Integration of community input
How well did the team do in considering input from the community, and putting processes in place to analyze and integrate this input?
- Operational smoothness
How well have testnet, upgrades, and general operation been going on the Cosmos Hub and consumer chains?
Overall performance
How well is the team doing overall? Grades for each of the four categories should factor into this (i.e., If all four categories are ‘needs improvement’ then the overall performance should logically not be ‘exceeds expectations’). Two consecutive quarters of ‘needs improvement’ should result in a signaling proposal to remove the team.
