Introduction
Since November 2023, Hypha has been administering the Testnet Incentive Program – a pilot program to reward validator participation on the Hub’s ICS testnets and ease the economics of Replicated Security on the Hub’s active set.
This pilot program ran from November 2023 - April 2024 and was funded by the Atom Accelerator DAO. We’ve now reached the end of the pilot program and have had the funding renewed for another ~6 months with some modifications to incentive criteria and expectations for validators based on how Interchain Security is changing.
Pilot program transparency
AADAO’s initial funding was made in ATOM, but TIP was always denominated in USD ($100 - $500 USD for each period).
Periods generally lasted 5 weeks and contained 3 - 4 testnet events and at least one mainnet event (not tracked as part of TIP, but we don’t overlap testnet and mainnet operations).
| Period | Events | Total validators | Validators paid | Qualifying validators | Funds disbursed |
|---|---|---|---|---|---|
| 1 (Nov 22 - Dec 20) | 2 | 44 | 19 | 43% | 8000 |
| 2 (Dec 21 - Jan 24) | 4 | 51 | 16 | 31% | 7600 |
| 3 (Jan 29 - Feb 28) | 3 | 48 | 20 | 42% | 9600 |
| 4 (Mar 6 - Apr 10) | 3 | 42 | 24 | 57% | 11600 |
| 5 (Apr 17 - May 12) | 4 | 42 | 29 | 69% | 11800 |
In general, the percentage of qualifying validators rose as we progressed through the pilot period. Some validators dropped out (all of them reached out to give feedback about why they weren’t continuing) but the remaining validators became more proficient in node operations and hitting task criteria.
The end result is a smaller set of testnet validators who are extremely high quality and able to take the skills they practice on testnet and apply them directly to mainnet operations, often leading the set in upgrades and catching issues on the Hub.
Highlights
With roughly 45 validators participating throughout TIP, everything we demo and train on the ICS testnet gets taken to mainnet by about 25% of the Hub’s active set.
While we’ve always found it important to run through mainnet upgrades on testnet to see the code hitting a production environment for the first time, some of our most valuable events in the last six months have been about developing a shared understanding about the technology with the Hub’s active set:
- Skipping an upgrade that cannot be skipped via governance
- Preparing for a major state migration included in the v15 upgrade
- Reviewing the importance of consumer key assignment
- Preparing for Replicated Security chains converting to Top N = 95% in v17
Feedback
We’ve gathered a ton of feedback about testnet operations during this pilot period and made several changes to the program.
| Feedback | Adjustment |
|---|---|
| Event timing (always 14:00 - 16:00 UTC on Wednesdays) is a bad time for many operators in a variety of timezones. | Unfortunately this is the nature of our decentralized work – no time works for everyone and all we can do is set consistent expectations. Hypha is based in Eastern time and will continue to use this time slot. |
| Registration process involving mainnet keys is not accessible for validators with really strict security practices for mainnet keys. | We’ve adjusted to allowing emails from a validator’s security contact to count as proof, but you’ll need to contact Hypha personally to allow that option. Our preference is still to use keys directly. |
| A requirement to sign within 5 blocks of an upgrade is far stricter on testnet than on mainnet because Hypha’s validators ultimately control how quickly an upgrade occurs. | To address this, we start Hypha’s validators 15 minutes after the upgrade height. |
| Days with multiple TIP events (i.e., one Wednesday might have 4 different tasks to be completed) mean that even validators who are paying attention to the event might make a mistake and then miss out on an entire period’s rewards. | The goal of the program is to have people regularly present on the testnet. Going forward, we will only require validators to complete one task per Testnet Wednesday to remain eligible for incentives. |
We have also received feedback about our testnet requirements being too demanding in general.
The testnet is not a place to take it easy.
Testnet is the place to see where everything might go wrong and how we might recover.
This is where we see the bugs, the resource intensive operations, the huge state migrations. Testnet machines and operators need to be at least as reliable as on mainnet so that we can handle all the stuff that we hopefully never need to handle on mainnet.
Validator economics
TIP was initiated with two goals:
- Ensure that the Hub has a thriving, production-quality testnet
- Ease the economics of Replicated Security on the Hub’s active set
The economics of Interchain Security have changed drastically since TIP first came on the scene. Neutron’s Proposal #867 directly compensates validators with 10k NTRN for running Neutron nodes, which is a style of compensation that may become the norm for any new Top N chains on the Hub.
Interchain Security is also on the cusp of transitioning to ICS 2.0, allowing for both Top N and opt-in chains. Opt-in chains will not mandate any validators to run them – validators will be able to choose solely based on what makes sense for their own business.
This reduces the need to compensate validators for general work of Interchain Security, though not the need for a production-quality testnet.
The new TIP
TIP has been renewed with a 75k USDC grant from AADAO through to the end of October (or until the funding runs out if we have enough validators)!
Our criteria set is:
- Criteria 1: Be an active mainnet validator. Submit proof via our Google form. If you are not able to use mainnet keys for proof, contact me @lexaMichaelides on Telegram or Discord, or email lexa@hypha.coop
- Criteria 2: Validate all available consumer chains secured by the Cosmos Hub (Neutron and Stride) using assigned consumer keys! Reused Hub keys will not be accepted!
- Criteria 3: Remain unjailed on the provider chain for the entire period.
- Criteria 4: Participate in every Testnet Wednesday event according to event criteria (e.g. sign within five blocks of a major upgrade, sign by a designated block after an upgrade).
- Criteria 5: Run a testnet infrastructure setup that closely mimics mainnet.
Expect to see more lenient event tasks for criteria 4 as well!
These date are subject to change:
| Period | Start | End |
|---|---|---|
| Period 6 | May 29 | Jul 3 |
| Period 7 | Jul 10 | Aug 14 |
| Period 8 | Aug 21 | Sep 25 |
| Period 9 | Oct 2 | Nov 6 |
Join the Interchain Security testnet!
- Join the testnet as a validator: instructions
- Fill out our Google form to register for TIP
- Make sure you are part of the Interchain Discord and have been assigned the
@Interchain Securityrole - Follow the #testnet-announcements in Discord and our testnet schedule to know when events are coming up