Introducing the testnet incentive program

ATOM Economic Zone
1

Introduction

Hypha Worker Co-op currently stewards the Cosmos Hub testnet program, which was established over the course of 2023 and now includes two live gaia testnets, multiple consumer chain testnets, and regular Testnet Wednesday launch/upgrade events. In the coming months, we intend to build out the testnet program’s capacity for chaos and stress testing with the introduction of Wargame Thursdays, which will put the network to the test with production grade security attacks and testing.

Testing for Interchain Security

Over 2023, Interchain Security has become the core product offering on the Hub, yet its economics remain unsustainable for validators who provide the infrastructure required to support consumer chains.

Like all networks, the Replicated Security testnet depends on the quality of its validator set. As a testing platform for the Cosmos Hub, it especially depends on validators who bring their Hub mainnet infrastructure and skills to the testing environment for both rehearsals and stress testing. Chaos testing and wargaming, in particular, benefit from a robust and highly participatory testnet as we seek to emulate mainnet conditions and see how the Hub will fare under various attacks and tests that seek to exploit potential vulnerabilities.

As validators run a new node for each additional consumer chain, participation in testnets and even mainnet becomes more and more financially challenging, particularly for the lower portion of the active set.

Introducing the testnet incentive program

As part of a 6 month pilot funded by a grant from the ATOM Accelerator DAO, Hypha is launching a testnet incentive program to encourage validators to create a production-grade testing environment on the Replicated Security testnet.

Beginning in Nov/Dec 2023, eligible validator operators can be compensated between with ATOM worth between $100 and $500 USD per monthlong period depending on their mainnet voting power. Our first period will be from November 22, 2023 to December 20, 2023.

Scope and reach

This incentive program applies only to the Cosmos Hub Replicated Security testnet. It is not responsible for incentivizing other testnets, with the exception of consumer chain networks that are secured by the Cosmos Hub provider chain.

This pilot program is intended to fund work only for 6 months: November 2023 - April 2024. In this time, we expect the Hub team at Informal Systems to continue researching technical solutions to improve Interchain Security and make the technology more affordable for validators. Several potential solutions are already under discussion in the Cosmos Hub forum: Atomic IBC with Megablocks and Partial Set Security.

Funding allocation

For a 6 month pilot period, the testnet incentive program is funded by a 50k USD grant from the ATOM Accelerator DAO, 90% of which will be paid directly to validators. The remaining 10% will cover administrative overhead for the validator members who are managing a multisig wallet and sending out monthly payments (listed below).

Individual validators will be incentivized according to their mainnet voting power.

  • All validators will be eligible a baseline payment of $100 USD per month.
  • Validators in the middle 33% by voting power will be eligible for an additional $200 USD for a total of $300 USD per month.
  • Validators in the lowest 34% by voting power will be eligible for an additional $400 USD for a total of $500 USD per month.

Payments will be sent from a 2/3 multisig (cosmos1078gmgntta9qguve7c52dhxrs45xumjvcwrutd) of representatives from:

  • Silk Nodes
  • Stakecito
  • Hypha

Any unused funding will be returned to the AADAO wallet at the end of the funding period.

Incentive criteria

To be eligible to receive payment for a particular period, validators must meet the following criteria. Proof for mainnet must be submitted via this Google form.

We update our schedule with new Testnet Wednesday events and indicate which events will be part of which incentive period (roughly 1 month per period).

:warning: Our first period will be from November 22, 2023 to December 20, 2023.

Network Criteria Proof
Mainnet Be an active mainnet validator on the Cosmos Hub. Mainnet validator must send a tx with a memo containing to cosmosvaloper address of their testnet validator and submit proof via Google form.
Mainnet Validate all available consumer chains secured by the Cosmos Hub. Hypha checks mainnet validators and blocks to confirm.
Testnet Remain unjailed for the duration of the period on the provider chain. Hypha testnet recorder reports jailing events.
Testnet Sign within five blocks after every scheduled Testnet Wednesday event in the period (consumer chain launch, chain upgrade, chain re-launch, etc). Hypha testnet recorder reports on block signing during and after events on Wednesdays.
Testnet Run mainnet-grade infrastructure (nodes, sentries, etc) on all chains. Disclosure from operators via Google form.

Join the Replicated Security testnet!

Here are the joining instructions for the Replicated Security provider testnet.

Coordination for the testnet happens on the Cosmos Network Discord, #replicated-security-testnet channel. Join here and ask for the Replicated Security role.

Need more info? Get in touch with @lexaMichaelides on Telegram or Discord.

9 Likes
2

To clarify, this is by voting power not rank. At this point in time, there are 7 validators in the top 33%, 15 validators in the middle 33%, and 155 validators in the bottom 33%

3

Looking forward to the Testnet Wednesdays.

Could you check the google form? Looks like we dont have access right now.

1 Like
4

Should be fixed. Thanks for catching that!

5

I’m really glad AADAO was able to fund this initiative from the Hypha team. Incentivizing validators to ‘show up’ and do crucial prep work in prevision of Mainnet is a no brainer, especially in the context of Replicated Security which is a more costly process.

6

The testnet incentive program is an interesting idea. Several months ago you suggested with @ala.tusz.am the UBI for validators running all consumer chains as an incentive to run consumer chains (https://forum.cosmos.network/t/scaling-up-with-conditional-basic-income). Why not fund a similar incentive program for this validator UBI? It could be tiered also with these 33% ranges for example. What do you think about this @jtremback @Youssef?

2 Likes
7

I had always intended Conditional Basic Income as a stopgap measure while waiting for further developments in cost reductions (which are now underway, as Jehan’s team researches things like atomic ibc and partial set security; and folks discuss new economic standards for consumer chains that might be coming to the Hub).

As long as we are doing a stopgap measure, I would rather get two birds with one stone and improve our testing ability, since this will also improve whatever new features are developed for ICS.

2 Likes
8

Hi @Cosmic_Validator

Thanks for the thoughtful feedback. I am in agreement with @lexa

I see conditional basic income (or universal basic income) as a temporary measure while Informal works on improving the current model to reduce the cost to run Replicated Security chains.

On the testnet incentive program above, we see it as a pilot from AADAO’s perspective and a step in the right direction. This pilot could potentially be renewed (and why not improved on) for a longer period and a larger budget once we hear back feedback from Hypha and the community at large.

Again, the best way to address costs of running consumer chains is to fix the root cause (which is closely tied to the inherent design)

1 Like
9

hey @lexa can new validators join the testnet program?

10

Certainly, we want as many validators as possible. Feel free to message me about it.

1 Like
11

Sure where can i connect with you to get some more details , here or discord?

12

Telegram or Discord. I’m @lexamichaelides on both.

1 Like
13

Greetings, I think that program is a great idea!

One thing that is unclear: can you clarify the requirements for this part?

Run mainnet-grade infrastructure (nodes, sentries, etc) on all chains.

My concern is that my servers at home are fully utilised and there’s no way to run the testnet and all its consumer chains at home, so I was thinking of running everything at cloud instead, but the rest should stay the same (KMS, monitoring, security patterns etc.), so I wonder if that would make me eligible.

Also, would be nice to hear what is the list of the requirements for infra to be eligible in general.

1 Like
14

We can’t reliably detect issues that might occur on mainnet unless validators are using their same mainnet infrastructure. The thing we are trying to avoid is having everything go well on testnet and then realize that we really hadn’t duplicated mainnet at all because mainnet involves hardware setups but testnet is entirely in the cloud. This is particularly important for wargaming, where we want to attack the testnet and see exactly what we might face if the same attack is performed on mainnet.

No, running things in the cloud does not meet the ‘mainnet-grade infrastructure’ requirement if you are not doing the same on mainnet. If you are operating bare metal on mainnet, you need to do the same for testnet to be eligible.

There are no hard requirements for eligible infrastructure aside from being the same as what you use on mainnet.

2 Likes
15

There have been some concerns about associating a mainnet validator key with a testnet validator key, so I’m introducing another way to verify that the team applying for the incentive is the same team operating the mainnet validator:

On mainnet Cosmos Hub, send a trivial amount of ATOM (e.g. 1 uatom) from your validator address to the multisig address cosmos1078gmgntta9qguve7c52dhxrs45xumjvcwrutd.

The memo field of this transaction must contain the cosmosvaloper address of your testnet validator.

If you have already submitted proof via the form using the delegate tx method, there is no need to submit proof again. Either of these methods is acceptable, but we recommend this mainnet-only one as it is simpler. The Google form and top-level post have been updated to reflect this.

I appreciate everyone’s feedback and patience as we work out the bugs in this pilot program!

2 Likes
16

Today marks the end of Period 1 (Nov 22 - Dec 20) of the pilot program! Over the last month, Hypha has tracked participating validators’ eligibility and refined the criteria based on feedback.

Criteria

  1. Be an active mainnet validator. Submit proof via our Google form
  2. Validate all available consumer chains secured by the Cosmos Hub (Neutron and Stride).
  3. Remain unjailed on the provider chain for the entire period (tracking available here).
  4. Participate in every Testnet Wednesday event according to event criteria (e.g. sign within five blocks of a major upgrade, sign by a designated block after an upgrade) (tracking available here).
  5. Run a testnet infrastructure setup that closely mimics mainnet.

Note – for Criteria 5, the following differences were counted as not meeting the criteria:

  • Disclosure via the Google form of “I am running different infrastructure or opsec on testnet and mainnet”
  • Running mainnet nodes on dedicated machines but running all testnet nodes on the same machine (including with containers)
  • Running mainnet nodes on bare metal but testnet nodes on VMs

These decisions are made in the name of keeping validator nodes on the testnet as close to mainnet as possible for use in stress-testing and providing a prod-level testing environment.

Participation and payouts

Overall, we had 44 validators participate in period 1, including some brand new to the Hub active set. Together, we tested the upgrade to v14, v14.1, and Neutron’s biggest upgrade since launch.

Criteria Pass Fail
1: In the active set 43 1
2: Validates all consumer chains 42 2
3: Unjailed for entire period 29 15
4: Participated in all events 29 15
5: Running mainnet-grade infra 39 5
Eligible for payment 19 25

Of the $50k USD grant, $8k was disbursed this period along with $2.5k in administrative fees to the multisig handlers (Bosco from Silk Nodes and Ghazni from Stakecito). All transactions are viewable from the multisig account wallet and emails have been sent out to all participating validators with more details.

The next period will take place from Dec 21 - Jan 24 and all failing criteria will be reset, giving new validators and period-1-ineligible validators a chance to become eligible for remaining periods.

What to do if you were ineligible this period?

If you are joining the program, let us know using the Google form!

If you are already in the program. check your email to see which criteria you didn’t hit.

  • Make sure you are running Stride and Neutron (criteria 2)
  • Make sure your testnet alerting system will warn you before you get jailed (criteria 3)
  • Show up to all Testnet Wednesday events, announced in Github and Discord (criteria 4)
  • Make sure your testnet infrastructure matches your mainnet setup according to criteria 5

Many thanks to AADAO for funding this pilot program! We look forward to seeing all the validators again in period 2: Dec 21 - Jan 24

3 Likes
17

Yesterday was the end of Period 2 (Dec 21 - Jan 24) of the pilot program.

This was an eventful period with two upgrades (one of which was an emergency immediately following Neutron’s smooth mainnet emergency upgrade), one chain launch, and our first-ever game day!

Overall, we had 51 validators participate in period 2 (although around 4 are no longer on the testnet – maybe they’ve left without saying goodbye).

Criteria Pass Fail
1: In the active set 43 1
2: Validators all consumer chains 42 2
3: Unjailed for entire period 26 18
4: Participated in all events 29 15
5: Running mainnet-grade infra 39 5
Eligible for payment 16 28

$7.6k USD was disbursed this period, equating to roughly 832 ATOM. All transactions are viewable from the multisig account wallet and emails have been sent out to all participating validators with more details.

What to do if you were ineligible this period?

  • If you are joining the program, let us know using the Google form!
  • If you are already in the program. check your email to see which criteria you didn’t hit.
    • Make sure you are running Stride and Neutron (criteria 2)
    • Make sure your testnet alerting system will warn you before you get jailed (criteria 3)
    • Show up to all Testnet Wednesday events, announced in Github and Discord (criteria 4)
    • Make sure your testnet infrastructure matches your mainnet setup according to criteria 5

Major feedback this period has been about timing and communication which events are TIP criteria.

During a scheduled Testnet Wednesday, new announcements may arise during the allotted period (15:00 - 17:00 UTC) and adjust TIP criteria based on new information. Announcements made during a Testnet Wednesday period by tagging @Replicated Security must be carefully followed to ensure TIP eligibility.

Unfortunately we cannot pick a time slot that works for everyone in this decentralized space. While we could choose to rotate the time period, we think this would make it much harder for participants to keep track of when to show up for events.

Other considerations for the future of TIP beyond the pilot program include allowing for a percentage of missed events to allow some flexibility for operators with a good track record who miss only one testnet event in a period. This adjustment is not possible during the pilot program but will be considered if the program is renewed past the pilot period.

The next period will take place from Jan 29 - Feb 28. We’re taking a short pause between periods to let jailed validators unjail and give some breathing room before starting up again :slight_smile:

3 Likes
18

This week closed out Period 3 (Jan 29 - Feb 28) of our pilot program.

We only had three official Testnet Wednesdays this period (one event moved to a Thursday so it didn’t count as a TIP criteria) but fit in nine different criteria.

We’re down to 48 validators participating in the program (some have confirmed that they’ve left, others have joined).

The feedback from validators who have left is largely that it’s still not financially viable to participate on the testnet if they must hit every criteria in order to get the incentive. This feedback will be incorporated into future iterations of this program if approved by AADAO, but will not change the criteria for the pilot program.

Criteria Pass Fail
1: In the active set 46 2
2: Validators all consumer chains 45 3
3: Unjailed for entire period 44 4
4: Participated in all events 21 27
5: Running mainnet-grade infra 44 4
Eligible for payment 20 28

$9.6k USD was disbursed this period, equating to roughly 818ATOM. All transactions are viewable from the multisig account wallet and emails have been sent out to all participating validators with more details. The account wallet has 2,681ATOM remaining, roughly $30k USD.

Other feedback from this period has included a new way to verify validator identity. Thanks for Cosmostation for this tip! Validators can update their security contact on-chain (an action that requires access to the validator keys to edit the validator.json file). View more information on this in the SDK documentation here.

The security contact can be queried using /cosmos/staking/v1beta1/validators.

The validator’s security contact email is tacitly approved by the on-chain keys to act on behalf of the validator. We will accept this as a method of verification going forward.

Next period will be March 6 - April 10. Again – good to take a little break between periods to unjail :slight_smile:

2 Likes
19

Period 4 ended this week.

We’re getting to the end of the pilot period, which also means that we’ll soon be able to review our criteria and make adjustments based on feedback from validators. Many validators have offered us input on what makes the program valuable, what makes it a pain to participate in, and what would make it more valuable for less pain (ideal).

Results this period were really promising – nearly all validators are meeting all criteria except the most challenging and changeable one – hitting every Testnet Wednesday event.

Criteria Pass Fail
1: In the active set 42 0
2: Validators all consumer chains 42 0
3: Unjailed for entire period 41 1
4: Participated in all events 24 18
5: Running mainnet-grade infra 41 1
Eligible for payment 24 18

We paid out $11.6k USD this period (1084 ATOM) and transactions are viewable from the multisig account wallet . Emails have been sent out to all participating validators with more details.

Last period, I reported that we had roughly $30k USD remaining. I have one correction to make here:

A total of $2.5k USD is still owed to the other two members of the disbursement multisig for their hard work and responsiveness in paying out many transactions each month (and razzing me for being late getting the list ready :triumph: ).

After period 4 payments and accounting for money owed to the multisig members, we have roughly $14.5k USD remaining. Depending on token values and participation, it’s possible that our pilot period will end after period 5 and any remaining funds will either be returned to AADAO or put towards an ongoing TIP system.

Next period will be April 17 - May 15.

2 Likes
20

In the midst of touring ISLE, period 5 of the Testnet Incentives Program has come to an end,

Criteria Pass Fail
1: In the active set 42 0
2: Validators all consumer chains 42 0
3: Unjailed for entire period 35 6
4: Participated in all events 30 12
5: Running mainnet-grade infra 41 1
Eligible for payment 29 13

This is our highest number of eligible validators yet – practice makes perfect and many of these validators have been with us since before TIP was even happening!

We’ve paid out the entirety of our remaining 1596 ATOM from AADAO this period. Due to a low ATOM price, it unfortunately meant that validators only received about 85% of the expected amount.

TIP has been renewed by AADAO with different eligibility criteria for ongoing use, so I’ll be back on the forum to report on our findings from this pilot period, our new criteria, and what the Hub’s ICS testnet will look like going forward.

Right now, we are still running the Interchain Security Lightning Experiment :desert_island: :zap: on the testnet and it’s not too late to join and earn incentives for testing and training on ICS 2.0!

1 Like