Introduction
This is an update for Proposal #839 (Fund 2024 Hub development by Informal Systems and Hypha Worker Co-op). Hypha and Informal Systems met with the 2024 Oversight Committee last week and are now ready to present reports and grades.
The 2024 Oversight Committee consists of:
- Stride contributor: Aidan Salzmann
- Neutron contributor: Avril Dutheil
- Polkachu representative: Polkachu
- Jim Parillo at Figment Capital
- Shane Vitarana at Stargaze
- James Hinck, Product Manager at Circle
This is a summary of Hypha’s portion of our 2024 Q2 Wrap-Up/Q3 Kick-off meeting which took place on July 18, 2024.
All members of the oversight committee have reviewed and approved this document prior to posting. Informal Systems has shared their report here.
Navigation
(I’ll add links later maybe)
Work done in previous quarter (Q2 2024: Apr 2024 - Jun 2024)
HIGHLIGHTS
ISLE testnet
- Testnet was originally planned to take 4 weeks
- Rapid acceleration of ICS 2.0 launch plan to hit the **June 5 mainnet date **
- Launched two Top N chains
- Launched three opt-in chains, including Elys Network
- Elys Network team ran communications for their launch with technical support from High Stakes and CryptoCrew
- Hub mainnet upgrade to v16 in the middle of the week
- 25% of Hub’s validator set was involved
- Quokka Stake identified a Hermes evidence submission bug, resulting in Hermes v1.8.3
- Grant received, paid out to validators, and excess funding returned in 2 weeks
- Unanimously positive feedback from involved validators (below, see survey results from 12/45)
- 5 = strongly agree
- 4 = agree
Validator training for Neutron and Stride conversion to Top N
- Observed node behaviour during ICS 2.0 devnet and
providerv17 upgrade - Wrote and distributed information to mainnet set to prepare for conversion (for example, adding content to Informal’s mainnet comms)
- On-call support during mainnet v17 upgrade to inform validators
Power shaping demonstrations
- Exploring power shaping parameters in ICS 2.0
- Lots of interest from prospective consumer chain teams and validators
- Validator demo days for set cap, power cap, allowlist/denylist
TIP report and renewal
- Over the pilot program, percentage of qualifying validators rose from ~40% to ~70%
- Renewed funding for 6 more months
- Ongoing investigation into how to continue funding testnet participation more efficiently (e.g., with delegations) as opposed to direct payment
CHALLENGES
Governance-coordinated software upgrades swapping binaries
- Increases the comms friction with validators
- Introduces churn into the testnet process
- Testnet upgrade path deviates from mainnet (e.g., v16.0 → v17.0 for testnet, v16.0 → v17.1 for mainnet)
- Per-upgrade coordination with Informal Hub team on the appropriate upgrade testing
- Scheduled v17.0 upgrade actually used v17.1 binary
- Voting: May 17 - 31
- Binary swap announced: May 31
- Upgrade: June 5
- Scheduled v18.0 upgrade actually used v18.1 binary
- Voting: July 1 - 15
- Binary swap announced: July 9
- Upgrade: July 17
- Expedited software proposals (live as of v18) will help by cutting the voting period down to 1 week instead of 2
v17.1 chain halt
- 4 hours of getting the chain back online (great coworking and communication though)
- Postmortem by Marius here
- Revealed weaknesses in the testnet’s realism and ability to catch bugs/issues that come up under mainnet conditions (e.g. full validator set with validators moving between active/inactive)
- Testnet validators are now too well-trained for good testing
- Need to incentivize ‘poor’ behaviour or allow validators to take risks without removing their funding
- Good feedback from validators (especially Quokka Stake) about how to do this.
v17.3 emergency upgrade
- Emergency upgrade to address a security vulnerability
- Hands-on coordination with validators
- Continued collaboration with Informal Systems and Amulet about improving the emergency upgrade process
Lorenzo dropping out of ISLE
- We had hoped to launch two real consumer chains during ISLE
- Lorenzo team backed out due to not being technically or socially ready for launch
- We can’t depend on incoming consumer chains to truly be ready to fit into the Hub’s timelines
- Especially new chains (as opposed to sovereign transition chains) need an incredible amount of support
ADJUSTMENTS
Planned and executed
In progress
Planned and delayed
Planned but removed
Added
Upgrade and feature testing
- Devnet for ICS 2.0
- Fresh, stateful testing for v16, v17
- Feature testing for ICA controller, rate limiter, fee module
- Feature testing for ICS epochs
- Feature testing for ICS 2.0 power shaping (power cap, set cap, allow/denylists)
- Testnet upgrades for v16, v17
- Transition testing for consumer chains becoming Top N
- Expanding test coverage of gaia to include basic use cases (e.g., try sending every transaction, try submitting proposals of varying length, try running out of gas)
- Fresh, stateful testing for v18
- Feature testing for CosmWasm integration, fee market, expedited proposals
Testnet Incentives Program
- Report on TIP pilot program
- Renewed funding from AADAO through to ~November
- Adjusted eligibility criteria based on feedback
- One game or demo day per month
- ISLE testnet
Gaia Maintenance and Interchain Security
- Major support on v17 mainnet upgrade comms
- Launch assistance for consumer chains (no chains are ready for mainnet)
- Third party module and release integration (CosmWasm wasn’t on mainnet yet)
- Support on testing CosmWasm integration
- Major support on testing feemarket integration
- Testnet support for permissioned CosmWasm (CosmWasm wasn’t on mainnet yet)
Other work
- Features & fixes upstreamed to interchaintest, including full support for launching ICS chains (it used to only work for certain versions of certain chains)
Q2 OPERATIONAL REPORT
- Upgrades:
- Governance: v16, v17.1, v18.1
- Coordinated: v17.2
- Emergency: v17.3
- Consumers:
- Neutron v3.02, v3.05, v4.0.1 (with Slinky)
- Neutron emergency upgrade (v3.06)
- Stride v22
- Elys launch on ISLE testnet
- Security events: v17.3 emergency
Q2 funding report
Publicly accessible funding report spreadsheet: [Public] Hypha Q2 financial reports for Cosmos Hub - Google Sheets
325k USD is a single FTE billable rate for the year. At $1.2M, that means Hypha’s fee cap allows us to have a maximum** of 3.7 FTE. **FTE assumes that the person works 2080 hours per year, or 520 hours per quarter.
Percentage of fee cap used = total FTE / max FTE
USDC earned = percentage of fee cap used * USDC fee cap
ATOM earned = percentage of fee cap used * ATOM fee cap
Hypha wallets
|
||||
Budget wallet (vesting):
|
cosmos13gatuxfqxn7m3vtlydkyznfs9k8xv8283la5a5
|
|||
Available fee cap wallet:
|
cosmos1ax7krw2ymq8upaxwce7df3xpalpp0jddm29fws
|
|||
Payroll wallet:
|
cosmos1d4t7vv9500sma5sea7668pea93dyke8kpvdhyw
|
|||
Bonus wallet (vesting):
|
cosmos1s39x73k7l0s88jkm399nkdk693ljmkjd2x83es
|
|||
Burn rate
Remaining funds
_AFC _refers to our ‘Available Fee Cap’ wallet, where we make the entire monthly budget available to invoice based on what we have earned. Our payroll workflow involves transfers made each month, so we have compiled a cash flow report to clarify each transaction.
Plan for next quarter (Q3 2024: Jul 2024 - Sep 2024)
Major areas of work
Testing
- Usual testing: fresh and stateful genesis, feature testing, testnet upgrades
- Consumer chain launch rehearsals for sovereign-to-consumer transitioning chains
- Evmos
- Comdex
- Consumer chain launch rehearsals for new consumer chains
- Elys
- Contributing to testing in gaia repo
- Adding interchaintest workflows to gaia
Testnet Incentive Program
- Continued game days and demo days
- More thorough changelog reviews with recordings/published takeaways for mainnet use
- Looking ahead to include testnet participation in ICF delegation program
Gaia Maintenance and Interchain Security
- Release integration work – one of our engineers is taking this up
- Third party module integration for CosmWasm now that it’s on the Hub
- Testnet support for permissioned CosmWasm on the Hub
- Design and support for ICS Launchpad
Expected challenges
- Expedited software proposals reduces testnet timelines
- Mitigation: Continued close work with Informal
- Mitigation: Hypha cutting releases brings our awareness
Community input
Input received
- Survey after ISLE solicited feedback from testnet validators:
- Language adjustments to how we communicate TIP criteria
- Live feedback during TIP events in Discord:
- Language adjustments for announcements
- Adjustments to testnet criteria to facilitate more realistic validator behaviour
- Feedback from incoming consumer chains about lack of clarity in how to solicit validator interest based on Discord structures set up last quarter
Interpretation and incorporation of input
- Adjustments to testnet criteria ⇾ focus on education over testing (or ‘training’)
- Consumer chain feedback ⇾ focus on ICS launchpad support and design
Oversight committee meeting and grades summary:
Please note that an adjustment has been made to the definition of the categories that the committee uses to grade Hypha and Informal:
- Execution: How well has the team been executing?
- Direction: Is the team currently planning and working in the right direction?
- Feedback: How well did the team do in considering input from the community, and putting processes in place to analyze and integrate this input?
- Smoothness: How well ahve the testnet, upgrades, and general operation been going on the Cosmos Hub and consumer chains?
- Overall: How well is the team doing overall?
Hypha’s grades
- Execution: Exceeds expectations. Hypha successfully adjusted its program’s timelines in response to Hub release cycles without impact on the quality of the testnet operations.
- Direction:
Meets expectations. - Feedback: Meets expectations.
- Smoothness: Exceeds expectations. Hypha has successfully maintained a high degree of operational smoothness despite the increasing number of testnet consumer chain releases being coordinated.
- Overall: Exceeds expectations.
The committee generally regarded Hypha’s performance as outstanding and reliable, as demonstrated by their excellent organization, general positive feedback from validators participating in Hypha-run programs and testnets, and their ability to adjust to the Hub’s general timelines. Hypha’s role corresponds moreso to a “run” function than a “change” function.