Hypha Oversight Meeting: Q1 Wrap-Up, Q2 Kickoff

[lexa]

Introduction

As part of Proposal #839 (Fund 2024 Hub development by Informal Systems and Hypha Worker Co-op ), Informal and Hypha will be meeting with their 2024 Oversight Committee each quarter to go over plans and reflections for the quarter.

The 2024 Oversight Committee consists of:

• Stride contributor: Aidan Salzmann
• Neutron contributor: Avril Dutheil
• Polkachu representative: Polkachu
• Jim Parillo at Figment Capital
• Shane Vitarana at Stargaze
• James Hinck, Product Manager at Circle

This is a summary of Hypha’s portion of our 2024 Q1 Wrap-Up/Q2 Kick-off meeting which took place on April 11, 2024. The time since then has been spent refining the reports, collecting grades from the committee, and finalizing financial reports after quarter-close.

All members of the oversight committee have reviewed and approved this document prior to posting. Informal Systems has shared their report here.

Navigation

(I do these for every huge post I make and I hope they are helpful but if they’re not, please god tell me so I don’t have to do them lol)

• Work done in previous quarter
  • Highlights
  • Challenges
  • Adjustments
  • Operational report
• Plan for next quarter
  • Major areas of work
  • Expected challenges
  • Community input
• Oversight committee grades and summary

Work done in previous quarter

(Q1 2024: Jan 2024 - Mar 2024)

HIGHLIGHTS

Game and demo days on testnet

• First game day was very smooth (perfectly timed – a rarity in our industry)
• Topics covered in game and demo days this quarter:
  • Options for avoiding an upgrade (canceling, skipping) and worked with Informal to make a plan for the edge case of needing to skip a mainnet upgrade
  • Hub’s behaviour for proposals which don’t meet quorum – the deposit is burned just like when it’s vetoed.
  • Changelog review for v15 to raise awareness of param change proposal risks.
  • Refresher on assigning consumer keys and security contacts (another surprise tool to help us later!
• Consistently, ~35-40 validators participating in game days
• Lots of positive feedback collected in our update thread for Testnet Wednesdays: Citizen Cosmos, Quokka Stake

January 17, 2024: Aether testnet launch and mainnet support towards PSS

• 46 validators present for this launch
• Testnet launch went smoothly and highlighted a few concerns for validators (chain-id flag)
• Follow up bizdev with Informal led to Aether switching to a PSS launch plan

March 6, 2024: pion-1 upgrade with our new test engineer driving

• We hired a new team member for the testnets team to up our capacity
• Violet has onboarded very fast and ran her first event less than a month after joining Hypha

March 20, 2024: Mainnet upgrade to v15.1.0

• Early testing and collaboration with Informal led to reduced memory usage for a very resource intensive upgrade.
  • Our mainnet upgrade simulations on pre-release candidate builds used up to 100GB in memory. This led to adjustments in the migration code that cut the memory use in half.
• Rapid communication and pivot to get mainnet upgrading to v15.1.0 (which includes pfm patch) instead of v15.0.0
  • Our automated tests found consensus-breaking issues in the first release candidate for v15.1.0. This led to the removal of non-deterministic code in the following rc.
• Hypha ran our first mini-testnet on March 18 to test the mainnet upgrade path from v14.2.0 to v15.1.0 since both testnets were already running v15.0.0 (which does not have the pfm patch)
• Result is v15.1.0 hitting the Hub as a tested upgrade with experienced validators who gone through the upgrade path instead of an emergency upgrade or validators seeing it for the first time on upgrade day.
• 90% of validators who participate on the testnet were up and signing within 5 blocks for this upgrade. 50% of non-testnet validators were signing that quickly.

March 28, 2024: ICS 2.0 devnet launch

• Hypha is coordinating a small devnet with Informal, CryptoCrew, and polkachu.
• The chain launched on March 28 and we’re currently in the process of launching Top N chains and experimenting with opt-in and -out work.
• Learnings from devnet will feed directly into an incentivized testnet to prepare the Hub validators for PSS.

CHALLENGES

Multiple emergency/patch upgrades this quarter

• Neutron v2.0.1 on Jan 10 Pion v2.0.1 on Jan 10 squeezed in alongside Aether rehearsal
• Neutron v2.0.3 on Mar 3 no testnet emergency upgrade (v3.0.0 already scheduled for Mar 6)
• Hub v14.2.0 on Mar 7 testnet upgrade to v15.1.0 on Mar 27
• We don’t have a fixed process for emergency upgrades on testnet because so much depends on the timing of mainnet upgrade and the version that testnet is running (e.g, provider is already running v15.0.0 so we won’t downgrade to v14.2.0).
• Emergency upgrades are always disruptive – they introduce the need for additional upgrade events on testnet between all the scheduled ones.

Running TIP and keeping validators invested at this payment level

• Continued feedback about how the program is too strict on validator participation (e.g., missing 1 criteria makes them ineligible for the whole period)
• Trend is that either a validator will show up reliably for all events without being paid OR demand payment even when they don’t show up reliably.

Navigating norms around open source code

• Aether codebase originally wasn’t open-source
• Prop #93 established a norm that community-funded projects needed to be open-source
• TIP is community-funded by proxy (AADAO funded TIP, Community Pool funded AADAO)
• We decided to only incentivize launch participation if code became open-source in time (which it did)!

ADJUSTMENTS

✅ Planned and executed
⏳In progress
▶️ Planned and delayed
❌ Planned but removed
🎁 Added

Upgrade and feature testing

✅ Fresh genesis testing for v15.0.0
✅ Stateful genesis testing for v15.0.0
✅ Testnet upgrades for v15.0.0
✅ Feature testing for v15.0.0
🎁 Fresh genesis testing for v15.1.0
🎁 Stateful genesis testing for v15.1.0
🎁 Mini testnet for v15.1.0
▶️ All testing for v16
⏳ Partial Set Security test plan

Testnet Incentives Program

✅🎁 One game or demo day per month
▶️ Pay out the entirety of TIP grant (ATOM went up!)
⏳ Apply for renewed TIP
▶️ Report on TIP pilot program

Gaia Maintenance and Interchain Security

✅ Follow security support process for mempool investigation
✅ Comms work for Hub upgrades
✅ Present and supportive for security issues (pfm patch)
✅ Technical support for Aether launch rehearsal
⏳ Process documentation for Interchain Security
⏳ PSS devnet

Other work

🎁 Open source contributions to interchaintest. It can now test interchain security with hermes.

Q1 OPERATIONAL REPORT

• Upgrades: v15.1.0
• Consumers:
  • Neutron emergency upgrade (v2.0.1)
  • Neutron emergency upgrade (v2.0.3)
  • Stride v17, v18, v19
  • Aether launch on testnet
• Security events: pfm escrow issue

All upgrades were smooth from an executional standpoint. Good comms, good teamwork between relevant teams and with the validator set.

Q1 funding report

Publicly accessible funding report spreadsheet:

325k USD is a single FTE billable rate for the year. At $1.2M, that means Hypha’s fee cap allows us to have a maximum of 3.7 FTE. FTE assumes that the person works 2080 hours per year, or 520 hours per quarter.

Percentage of fee cap used = total FTE / max FTE

USDC earned = percentage of fee cap used * USDC fee cap

ATOM earned = percentage of fee cap used * ATOM fee cap

Hypha wallets
Budget wallet (vesting):		cosmos13gatuxfqxn7m3vtlydkyznfs9k8xv8283la5a5
Available fee cap wallet:		cosmos1ax7krw2ymq8upaxwce7df3xpalpp0jddm29fws
Payroll wallet:		cosmos1d4t7vv9500sma5sea7668pea93dyke8kpvdhyw
Bonus wallet (vesting):		cosmos1s39x73k7l0s88jkm399nkdk693ljmkjd2x83es

Burn rate

Remaining funds

AFC refers to our ‘Available Fee Cap’ wallet, where we make the entire monthly budget available to invoice based on what we have earned. Our payroll workflow involves transfers made each month, so we have compiled a cash flow report to clarify each transaction.

Plan for next quarter (Q2 2024: Apr 2024 - Jul 2024)

Major areas of work

We outlined details of our testing processes in our Q1 kickoff report. To save space, all of those still apply and we’ll just give bullet points of out intentions for this quarter.

Testing

• Fresh, stateful genesis testing for versions released this quarter
• Feature testing for versions released this quarter
• Testnet upgrades for versions released this quarter
• Devnet for Partial Set Security
  • Launched in the first week of April (polkachu is validating)
  • Launched a first top N chain
  • Working through instructions for opting in
  • Possible plans for testing the transition from Replicated Security to Top N
    • RS will continue to work the way it always has
    • RS with a 5% soft opt-out is functionally identical to a Top-95% chain
• Expanding on our automated testing
  • Finish migrating our tests to interchaintest
    • Will keep contributing to and battle-testing the framework
  • Expanding test coverage of gaia. Test every type of transaction.
  • Adding coverage for realistic use-cases that happen on mainnet
    • Some happen via social convention, e.g. the issue with proposal length
    • Some are common failure scenarios, e.g. people running out of gas on a transaction

Testnet Incentive Program

• Renewal for another 6 months with changes based on feedback
  • Using security contacts to verify identity
  • Allow flexibility in missing some testnet events:
    • For days with more than 1 event (e.g., game days with multiple tasks), only require the validator to hit one event to remain eligible
    • Allow 2 missed days per 6 month period
  • New criteria: Running assigned consumer keys on all Top N or RS testnet and mainnet chains (do not reuse keys!) We are using TIP eligibility criteria to directly incentivize good mainnet behavior.
• PSS incentivized testnet (detailed out below in Interchain Security section)

Gaia Maintenance and Interchain Security

• In Q1, we hired another engineer as planned. In Q2, we’ll be transitioning into maintenance work.
• Partial Set Security incentivized testnet
  • AADAO funded via the Testnet Incentives Program
  • Big marketing and education push
  • Involving validators, consumer chains, integrators
• Transition assistance for Neutron and Stride as required
• Launch assistance for Aether and Elys
• Release integration work
  • Currently, Informal cuts releases
  • Hypha will move to being able to cut releases in Q2
  • Also expands the Hub team’s collective working hours since releases can be cut and tested all in Eastern time instead of only being cut in Europe time
• Third party module integration
  • Keep track of our smaller third party modules and when they’re being upgraded
  • Put in PRs into gaia with the upgraded modules
  • Raise any larger issues to Informal or to the third party responsible for maintenance
  • E.g., CosmWasm integration work
• Testnet support for permissioned CosmWasm on the Hub
  • Smart contracts on the Hub will require a lot more testing to ensure the security of the Hub

Expected challenges

• Too many incoming consumer chains overbook Testnet Wednesday events
• Hub state grows to be extremely resource intensive on our servers
  • Mitigation: Focus testing efforts on code efficiency, recommendations to dev team to improve efficiency
  • Mitigation: Patch gaia during testing
  • Mitigation: Bigger servers…
• Lack of incoming consumer chains leads to few testnet events
  • Mitigation: Creating our own game days, partnering with other organizations and projects to run test events
• Changes to features shortly before release reduces testing timeline
  • Mitigation: Close communication with Informal dev team
  • Mitigation: Improvising speed and efficiency of automated testing suite

Community input

Input received

• We receive feedback on our testnet work every Wednesday and incorporate it into the next event.

Interpretation and incorporation of input

How has this feedback modified the work executed this quarter, next quarter’s plan, and the year’s roadmap?

• Adjustment to testnet wednesday
• Adjustments to TIP payout criteria
• Preparing to support CosmWasm on the Hub now that prop #895 has passed

Oversight committee meeting and grades summary:

Overall, the committee was mainly curious about delays in TIP reporting and renewal. These delays are because ATOM value rose and so our initial funding lasted longer than expected

Meeting last quarter’s plan
All committee members graded Hypha as ‘meets expectations’.

Progress on overall roadmap
All committee members graded Hypha as ‘meets expectations’.

Integration of community input
3/6 graded Hypha as ‘meets expectations’.
3/6 graded Hypha as ‘exceeds expectations’.

Operational smoothness
3/6 graded Hypha as ‘meets expectations’.
3/6 graded Hypha as ‘exceeds expectations’.

Overall grade
On average, Hypha’s grade is ‘meets expectations’.

[btruax]

I like your navigation links!

[Govmos]

First of all, kudos to the team for the amazing work performed. We made a public shout out to both Hypha/Informal teams in a separate post and to avoid clogging the forum we will avoid repeating ourselves a second time in this post, but rest assured that we support your work just as much as Informal’s!

@lexa

Partial Set Security incentivized testnet

• AADAO funded via the Testnet Incentives Program
• Big marketing and education push
• Involving validators, consumer chains, integrators

As this may come in the near-term, could you share more details with us regarding the expected timeline?

[lexa]

I posted about it yesterday, actually!

Announcing the ISLE Incentivized Testnet 🏝️

What: Interchain Security Lightning Experiment (ISLE) testnet When: May 13 - May 17, 2024 Who: All Hub mainnet validators are welcome Why: Check out ICS 2.0 AND earn $200-500 for one week of testing and training How: Join the Interchain Discord and ask to be given the Interchain Security role Welcome to the Interchain Security Lightning Experiment – ISLE for short ISLE is a lightning-fast testnet program to get ICS 2.0 to mainnet in record time. Past incentivized testnets…

We’ll get docs and joining instructions out next week but for now, the first step is just to join the ICS testnet: testnets/interchain-security at master · cosmos/testnets · GitHub

[Govmos]

This post was actually the next tab for me read! Unfortunately for me your Q1 wrap-up was earlier in the list ^^ Anyway thanks for info, I’ll pass it on to the operators and we’ll do our best to be among the participants.