Hypha Oversight Meeting: Q3 Wrap-Up, Q4 Kickoff

[lexa]

Introduction

This is an update for Proposal #839 (Fund 2024 Hub development by Informal Systems and Hypha Worker Co-op). Hypha and Informal Systems met with the 2024 Oversight Committee last week and are now ready to present reports and grades.

The 2024 Oversight Committee consists of:

• Stride contributor: Aidan Salzmann
• Neutron contributor: Avril Dutheil
• Polkachu representative: Polkachu
• Jim Parillo at Figment Capital
• Shane Vitarana at Stargaze
• James Hinck, Product Manager at Circle

This is a summary of Hypha’s portion of our 2024 Q3 Wrap-Up/Q4 Kick-off meeting which took place on October 16, 2024.

All members of the oversight committee have reviewed and approved this document prior to posting. Informal Systems has shared their report here.

Work done in previous quarter

(Q3 2024: Jul 2024 - Sep 2024)

HIGHLIGHTS

Comdex rehearsal

• First live test of the allowlist feature with validators
• Caught and resolved issues with naming and formatting of the consumer chain files
• Validator training and clarification on validator power cap led to Comdex making changes

DAO DAO testnet integration

• Clarified what “permissioned” CosmWasm should look like on the Hub (permissioned on upload vs instantiation)
• Worked with DAO DAO team to get the correct settings for the Hub and put it on mainnet
• Prepared explanations and clear descriptions of CosmWasm settings for Hub community

Devnet

• Live devnet with recent fork of the Hub
• Updates for alpha/beta versions of gaia during development
• Facilitates a better and faster testing process prior to cutting release candidates

CHALLENGES

v20 API changes impacting smaller integrators

• Most large integrators updated to accommodate API changes (some still incomplete)
• Some smaller ones were missed and ultimately chose not to support the Hub anymore
• Max provider consensus vs max validators

v19.2.0 emergency patch

• Severe ICS bug that could have impacted the Hub and all consumer chains
• Lack of clarity in communication with the later ⅔ of the validator set
• Postmortem by Lexa here
• On the bright side – severe bug mitigated with no chain halt

ADJUSTMENTS

• Planned and executed
• In progress
• Planned and delayed
• Planned but removed
• Added

Upgrade and feature testing

• Devnet for ICS (repo here)
• Fresh, stateful testing for v18, v19, v20
• Testnet upgrades for v18, v19, v20
• Verified Stride rewards distribution post v20 upgrade
• Feature testing for permissionless ICS, inactive validators
• interchaintest-backed testing in cosmos/gaia
• feemarket params change testing (forum post here)
• Overhauling automated test workflows, now based on user stories (expected for v22)

Testnet Incentives Program

• Game Day: Permissioned CosmWasm
• Demo Days: Consumer modification, feemarket
• Period 6 completion: $13.2k to 28 validators
• Period 7 completion: $16.4k to 34 validators
• Period 8 completion: $6.6k to 14 validators
• Thorough changelog reviews (permissionless demo day)
• Stake redistribution policy to mimic mainnet active/inactive set
• Working on AADAO-funded extension into 2025
• Exceptions policy for risky behaviour and experiments

Gaia Maintenance and Interchain Security

• Crypto Dungeon rehearsal
• Comdex rehearsal
• Testnet support for permissioned CosmWasm
• Evmos rehearsal

Other work

• Design and testing support for Forge

Q3 OPERATIONAL REPORT

• Upgrades:
  • Governance: v18.1, v19.1, v20
  • Emergency: v19.2
• Consumers:
  • Neutron v4,
  • Neutron emergency upgrades (v4.1.0, v4.2.0, v4.2.1, v4.2.3, v4.2.4)
  • Stride v23, v24
• Security events: v19.2 emergency

Q3 funding report

Publicly accessible funding report spreadsheet: [Public] Hypha Q3 financial reports for Cosmos Hub - Google Sheets

325k USD is a single FTE billable rate for the year. At $1.2M, that means Hypha’s fee cap allows us to have a maximum** of 3.7 FTE. **FTE assumes that the person works 2080 hours per year, or 520 hours per quarter.

Percentage of fee cap used = total FTE / max FTE

USDC earned = percentage of fee cap used * USDC fee cap

ATOM earned = percentage of fee cap used * ATOM fee cap

Hypha wallets
Budget wallet (vesting):		cosmos13gatuxfqxn7m3vtlydkyznfs9k8xv8283la5a5
Available fee cap wallet:		cosmos1ax7krw2ymq8upaxwce7df3xpalpp0jddm29fws
Payroll wallet:		cosmos1d4t7vv9500sma5sea7668pea93dyke8kpvdhyw
Bonus wallet (vesting):		cosmos1s39x73k7l0s88jkm399nkdk693ljmkjd2x83es
Holding wallet (vesting):		cosmos12hpvh4p54q7fvcnr3cvsz9trly92jurmdh74gx

We have added the “holding wallet” to our report. This contains the bonus ATOM we’ve earned and have put in a vesting wallet that completes its vesting period one year after the bonus was earned, as described in the original proposal. This lock-up period is meant to produce longer term incentive alignment than allowing a team to liquidate immediately.

Burn rate

Remaining funds

AFC refers to our ‘Available Fee Cap’ wallet, where we make the entire monthly budget available to invoice based on what we have earned. Our payroll workflow involves transfers made each month, so we have compiled a cash flow report to clarify each transaction.

Plan for next quarter (Q4 2024: Oct 2024 - Dec 2024)

Major areas of work

• Improving our user story validation testing of gaia and ICS
• Running the Gaia Devnet
• Major restructuring of the Testnet Incentives Program

Testing

• Usual testing: fresh and stateful genesis, feature testing, testnet upgrades
• User story testing for new versions
• Devnet maintenance and operation

Testnet Incentive Program

• Continued game days and demo days
• AADAO grant renewal
• Major shifts in calculating payment based on validator feedback
• Looking ahead to include testnet participation in ICF delegation program

Gaia Maintenance and Interchain Security

• Continued consumer chain rehearsals with Forge
• Validator and integrator education around permissionless ICS
• Assisting with communication norms for permissionless chains
• Sunsetting theta testnet

Expected challenges

• Lack of validator support in Forge
• Uncertainty around product leadership for the Hub in 2025 (possible product teams have been identified but none have publicly committed yet – it’s a decision that understandably takes time)
• Increased Hub and consumer chain activity on Wednesdays (scheduling collision with testnet events)
  • Considering switching to Testnet Tuesdays

Community input

Input received

1. Increased complaints from validators about not being eligible for TIP funding based on single mistakes during the period.
2. Questions about the style of testing given the kinds of bugs we have been experiencing.
3. Secure hashes should be used to verify emergency upgrade procedures when binaries are distributed via Telegram or Discord.

Interpretation and incorporation of input

1. For the next TIP grant, transition to a point-based payment system rather than all-or-nothing for each period. This is the method we have used for time-bound incentivized testnets such as Game of Chains and ISLE, and is a better prototype for a delegations-based incentive system in the future.
2. Clarification between dev team and Hypha to reduce duplicated efforts and ensure coverage over both verification and validation testing.
3. Playbook draft of how to use hashes to verify messages on Telegram or Discord:
   1. [ This message concerning the v19.2.0 gaia emergency upgrade was written and released by Lexa, Marius, and Brian. It is approved by Hypha Co-op and Informal Systems. The hash of this message (including the square brackets) has been published via company accounts on X (@HyphaDoesCosmos, @InformalSystems) as verification. To confirm for yourself, copy the text including square brackets and generate a SHA256 hash to see that it matches those published by company accounts. ]
   2. 1383a8d2a7fcc4189886dd2862691584f568fb6ab8578005a4c2c0dbc535ed7f

Oversight committee meeting and grades summary:

Please note that an adjustment has been made to the definition of the categories that the committee uses to grade Hypha and Informal:

• Execution: How well has the team been executing?
• Direction: Is the team currently planning and working in the right direction?
• Feedback: How well did the team do in considering input from the community, and putting processes in place to analyze and integrate this input?
• Smoothness: How well ahve the testnet, upgrades, and general operation been going on the Cosmos Hub and consumer chains?
• Overall: How well is the team doing overall?

Hypha’s grades

• Execution: Meets expectations.
• Direction: Meets expectations.
• Feedback: Exceeds expectations.
• Smoothness: Meets expectations.
• Overall: Meets expectations.

Additional feedback from Avril Dutheil:

• Hypha continues to demonstrate reliability and a great ability to take in feedback.
• The vast majority of committed items were delivered, with missed items being related to third party delays.
• The Gaia v19.2 upgrade was a pretty problematic failure of process (Security advisory ICS-2024-002 and Gaia v19.2.0 patch retrospective), but it was well addressed in the retrospective and it seems likely that the org will improve from this experience / not face the issue again.