Dear community and validators,
This forum post is a request for feedback and Proposal draft seeking community funding for a relayer fee-grant support system for IBC traffic on the CosmosHub.
This governance proposal addresses the urgent need for supporting IBC relayers on the Cosmos Hub following the recent gas fee increase. It proposes the establishment of a fee-grant support system, funded by the community, to cover gas fees for IBC relaying activities. The proposal outlines the critical role of relayers in maintaining the network’s interoperability and the negative impact of increased operational costs on their sustainability. The proposing team consisting of respected Cosmos Hub relayers suggests forming a multisig group to manage the distribution of fee-grants to vetted relayers, offering a temporary solution to prevent service degradation and ensure the continuity of efficient IBC operations on the Cosmos Hub.
What is a relayer and IBC:
The Cosmos ecosystem is known for its interoperability technology dubbed, the Inter-Blockchain Communication protocol (IBC) that uses light client verification systems to allow for trust-minimized asset and message transfers between networks. The operators of this trust-minimized bridge are permissionless actors called relayers, they listen for packets on chain-A and post verifications for those packets on chain-b. An IBC transfer consists of 3 separate transactions of which 2 are orchestrated by the relayers, this means they pay the gas for these transactions. In order to relay IBC packets these operators need to run a full node (with sufficient archive) of both networks and a separate relayer software instance that connects to these full nodes. There are several different relaying softwares with the most used being Hermes and RLY. The finality of a user’s IBC transaction is directly related to the number of relayers/instances searching for packets and the delay of these machines to their respective fullnodes. To deliver a fast-finality and redundant service it is important that there is redundancy in both the full nodes, the number of relayer instances and the number of wallets from which transactions are posted.
What is the problem at hand:
On the Cosmos hub IBC relaying is mostly seen and operating as a public good and a marketing/profiling activity. This means relayers (which are often validators) are operating the necessary infrastructure at a loss to help the network grow or grow the brand of their business. Although this is a well-known fact and a respected activity within the Cosmos there is a situation where this becomes unsustainable. We believe this point has been reached on the Cosmos hub with the most recent gas increase in proposal 843 to a base of 0.005uATOM, an increase by the factor of 5 compared to before. A median IBC TX costs ~450k gas units currently (ibc light client update + 1 packet acknowledgement/receive/timeout) coming to a cost of ~0.0025 per message type and therefore ~0.0045 ATOM (*~8.5 = $0.038) per user initiated IBC tx. Including ICA and ICS message types the Cosmos hub is doing at minimum over 400k IBC messages per month totalling >1800 ATOM spent by relayers per month (further analysis indicates a total cost of 50-75 ATOM per day ~1500-2000 ATOM per month: IBC Fees | zanglang | Flipside). Anecdotal evidence shows active relayers spending 3-4 ATOM per day (prior to 5x fee increase) with the most active ones spending north of 10-20 ATOM per day. With reducing ICF delegations across the board (due to Atom per point going down and Relayer/RPC contributions being capped at 5 points) and the fact that many Relayers are lower in the validator set it has become more unsustainable than ever. So although ICF delegations help incentivize relayers, income from these delegations doesn’t come close to covering the costs for both Infrastructure and gas fees. We are now seeing 4+ active relayers actively stop relaying by keeping their wallet empty, this hurts the experience of ATOM IBC with users not being able to transfer funds and ICS chains not getting proper security set provider updates.
** above image shows many active relayers resorting to holding 0 ATOM and thereby not relaying.
Why should the community care:
The Cosmos Hub is exactly that, a hub. It is the largest onboarding avenue for users into the Cosmos ecosystem and one of the most liquid tokens across IBC. With a bad IBC/relaying experience we degrade the service quality of one of the major use cases of the Cosmos hub. Increasing the gas fee, while necessary to prevent spam attacks, has priced out relayers to the point where one of the core functions of the hub cannot provide a high quality service anymore, something that hurts the branding and investability of the hub. Transactions take longer or timeout, users will feel stuck potentially even leading to swapping to other IBC assets so as to finalize whatever on/offboarding they were doing. Simultaneously the lack of relayer support creates potential attack avenues on the Cosmoshub consumer chains as they are not kept up to date with provider set updates. Considering the manifold Cosmoshub features that rely on reliable IBC functionality (ICS, ICA, token transfers), lasting downtime could have adverse effects on the overall image and usability of the cosmos hub and the ATOM token.
The proposed solution/remedy:
This proposal draft, which has been created in collaboration by various relayer teams, including Lavender Five, CryptoCrew, Crosnest, Cosmos Spaces, Architect Nodes and IcyCRO, is seeking 8,000 ATOM (3x 2000 + Safety margin) to be send to a ⅗ multisig operated by TBD** to cover ATOM gas fees for IBC relaying to and from the Cosmos Hub for a period of 3 months.
**We are seeking 3 community representatives to join the multisig along with 2 relayer representatives. Please reach out to @Ertemann on Telegram if you are interested in participating or indicate so in this forum post! - There will be no compensation for this role or any role performed by the proposers for that matter.
The multisig address will distribute fee-grants to well known and vetted IBC operators under a fair usage agreement. Any ATOM leftover after the 3 month period can be returned or will roll over into the next period of Relayer funding if agreed upon.
This proposal does not pay relayers any ATOM, it merely offsets their native ATOM fees. It does NOT pay for the required hardware, service hours, counterparty fees and any other associated costs.
This is intended as a first step proposal to remedy the complete lock-out of ATOM IBC we see happening right now.
Multisig is responsible for signing-off on feegrant “grant” transactions to individual wallet addresses. Initially, this may need to be done manually as most major relayers “sign up” to the system with their wallet addresses, but it should be possible to automate this process in the future to reduce operational overhead and multisig fatigue.
Each operator may sign up with multiple wallet addresses, and each address will be granted one feegrant of equal value with an “expiration date” set to the end of the proposed 3-month period. For now this means relayers will only be able to leverage the Hermes software until RLY allows for feegrant wallets external of the Software’s keychain.
The feegrant will have a “total spend limit” set that will halt any further expenditure when the wallet address has exhausted the number of ATOMs granted to them.
The feegrant will have a “period limit” set that specifies the maximum number of ATOM that can be spent in a time period, for example, 10 ATOM per 24 hours. This can be used to prevent malicious operators from willfully exhausting the ATOMs granted to them by setting incorrect gas prices.
The feegrant will have a limit of allowed message types: MsgRecvPacket, MsgAcknowledgement, MsgUpdateClient, MsgTimeout, MsgTimeoutOnClose, MsgChannelOpenTry, MsgChannelOpenConfirm, MsgChannelOpenAck
**official documentation for obtaining a feegrant from the multisig and the requirements will be posted in the coming week. Any feegrant recipient will have to apply publicly - we suggest to use a cosmos forum thread for that.
- What are the criteria for vetting operators, and how does one sign up?
Any operator is welcome to make a public application and go through the vetting process to gain access to the fee grant. However, any operator that has processed more than 500 IBC messages (~1% of total) in the past month can submit their application for expedited approval. The exact requirements and summarized fair use agreement will be posted shortly in the above mentioned documentation page. We would love any feedback on specific gating requirements the community deems important.
- Wouldn’t the upcoming SKIP block SDK decrease fees again based on usage only?
The effects of EIP-1559-style fee markets on relayers cannot yet be quantified very well. This proposal is seeking to reimburse relayers for the transaction fees they spend in the future. The implementation of the Block SDK will only influence the relayer burn rate, and may be considered as a factor in upcoming multisig replenishment proposals.
- Doesn’t the ICF pay relayers on the Cosmos hub in the form of a delegation?
While ICF delegations do acknowledge IBC relayer contributions, they do not suffice to cover relayer transaction fee expenses in light of the recent min fee increase. This proposal aims to work towards a more sustainable model, where fees spent by whitelisted relayers are covered through fee grants, and incentivization for relayer operation comes from ICF delegations.
- I thought my part of my transaction fee goes to relayers? If not, why not incl relay costs in tx fees"
This is not the case currently, the gas fees users pay is only to cover them posting the initiation of the IBC packet. To properly relay packets in between chains 2 relayer transactions are needed, one on the source- and one on the destination chain (“receive packet” and “acknowledge packet”). Any TX thereafter the gas is paid for by the relayer. The idea to add a fee to each transaction is good and has been written out as code in the ICS-29 standard.
- Why are relayers not pushing ICS-29? That should pay them pro-rata.
ICS-29, also known as “relayer-fee-middleware” aims to charge users for IBC transaction fees. In its current design, ICS-29 hasn’t gotten much adoption for a number of reasons. It is subject to MEV by validators misbehaving and it puts a too significant effort on the # of packets transferred and not the quality of service. It will cause a race to the bottom for relayers to the point where users pay more but relayers still don’t make a profit. Furthermore, currently only new channels can use this feature (as channel upgradeability is yet to be completed), which means that ISC-29 is no viable option for now. We would love to see this change in the future. More recent developments like permissioned IBC (ibc-sync), ABCI++ pre-block building and other solutions can help ICS-29 become better at its original goal. Additionally relayers are looking at a PFM-based fee that could be used to fund this fee-grant proposal into the future.
- Other networks that want to use ATOM should pay for Relayer contracts and fees themselves.
Frankly we agree, and to some extent this is happening already. CryptoCrew has an agreement with Neutron and the AADAO and certain providers also have private agreements with dedicated chains. In our fair use agreement we will request relayers to operate existing relaying contracts from a different address. We don’t think it is fair to assume that all chains and all relayers are able to pay for dedicated contracts or have gotten enough credit to obtain one. We hope this proposal will promote more relayers to start back up and new ones to join and learn in an environment where they are not simultaneously losing money. Over time this will only grow IBC and thereby the Cosmos Hubs relevancy.
- Wouldnt removing the Client update on all messages be a good way to reduce costs?
Yes, but there are some caveats here. First we need clients to be up-to-date enough to be able to verify the transaction between chains. This means we can only start reducing Client updates effectively by batching TXs. Due to the current relayer race this is not done effectively, by making more agreements together we could change this with this proposal. Secondly the RLY and Hermes software does not properly support Client checking yet, it most often defaults to always updating. This is something both Informal and Strangelove are working on to improve.
- How can we better leverage batching to reduce costs?
The feegant multisig will set a requirement on the RPC pull times and WS batching delays for all requestors. This should have minimal impact on finality for users but can efficiently increase the IBC capacity and reduce fees. Control will be based on the aforementioned fair usage agreement.
- How will you make sure relayers don’t empty the feegrants by using insane fee settings on their transactions?
Hard caps are set for each wallet that receives a feegrant so they may not spend more than a certain amount of ATOM within a fixed period of time. We will set up public analytics dashboards that will display the transactions and fees consumed so far by each wallet. Any one address that spends unusually more than its peers will stick out like a sore thumb, and malicious operators can be quickly identified and have their feegrants revoked.
- Are relayers profiting from this?
No, feegrants can only be used to spend on gas and relayers can not take these ATOMs into their own control. No funding is requested to compensate relayers for time and infrastructure at this time. It does reduce costs for relayers as they will no longer have to spend their own bought/earned ATOM - one could consider this a profit, or at least a reduced deficit.
- Why are you asking for such funding on the Hub only? Osmo and others increased fees too recently.
The Cosmos hub has always been the chain with one of the most IBC transactions and also one of the highest fees, this combination makes the Cosmos Hub significantly more expensive to relay for. With the recent increase of the globalfee minFee param this issue also seems to have come to a boiling point, as some relayers are shutting down shop for ATOM specifically. Other networks might have operation DAOs or foundations that create contracts with relayers. As such a possibility does not exist for the Cosmos hub, this proposal is a first step in improving the situation for IBC relayers on the hub.
- Won’t this just create another “cartel”? How would new relayers be able to get in?
Although the multisig will have 2 relayers on them the majority will be community members. All relayers will be welcome to apply for a feegrant as long as they can meet the predetermined requirements. Collective decision making around Relayer configuration can actually significantly improve efficiency and performance. This proposal will most likely remove some of this competition and “free market” principles for relayer operators, that is something voters should take into consideration for better or worse.
- Why is this even needed?
Seamless and trustless interoperability is one of the core USPs of the Cosmos hub. Immaculate user experience is paramount when it comes to adoption of the IBC protocol and the Atom Economic Zone. We believe it’s in the best interest of the Cosmos hub community to support relayer operators in providing the best-possible service quality by offsetting their transaction fees through the proposed infrastructure.
Ertemann - Lavender.Five Nodes
Claimens - CryptoCrew Validators
Tricky - Cosmos Spaces
Jerry - IcyCRO
Architect - Architect nodes
David - Crosnest
- Be a relayer on the hub and relay on channels to active and important counterparty chains**
- lum network
- gravity bridge
- Have > 10 IBC txs relayed for any of the channels connected to the listed chains in the past 30 days. For the baseline we use Map of zones
- The multisig will re-evaluate the Chains and thereby the grantees on a monthly basis
- Operation is allowed with the following relayer softwares: ts-relayer, go-relayer (RLY), rust-relayer (Hermes)
- relayer must set batch_delay / polling frequency of at least 500ms or more
- block time / batch delay = ~5s / 0.5s => max 10 txs per block per account
- if any account posts more than 10 txs into one block: misbehaviour
- packet efficiency > 10%
- Multiple wallets per entity are eligible > every account that is eligible within the outlined criteria
**relaying to other chains using the feegrant is allowed as well, its merely for qualification
- Apply via Github PR & provide relayer account address
- Automation checks eligibility
- Multisig confirms - Sends fee-grant message
- Review of eligibility on a monthly basis
A tier system will be followed for the total daily budget:
Low tier - 5 ATOM daily spend limit
High tier (top 10 accounts by effected packets confirmed) - 50 ATOM daily spend limit
The oversight committee (multisig members) reserves the right to evaluate and revoke whitelistings on a daily basis. Furthermore, the oversight committee has the right to evaluate and update the onboarding parameters if needed. Reasons for revocation include but are not restricted to: spamming (unnecessary or unnatural transactions), misbehavior against onboarding criteria (batch delay, efficiency), use of feegrants for non-IBC related messages, making use of misconfiguration by mistake of the multisig.
Clemens from CryptoCrew has created a repository for the relayer-dao (GitHub - clemensgg/ibc-relayer-dao) and has setup analytics and automation to help the multisig operate the feegrants.
- monitoring of feegrant state (prom exporter + grafana)
- relayer efficiency metrics (indexer + prom exporter + grafana)
- PR checking for onboarding
The oversight committee consists of:
Clemens (CryptoCrew): cosmos1705swa2kgn9pvancafzl254f63a3jda9ngdnc7
Ghazni (StakeCito): cosmos1qm5agp78atuf9pyalsq4w30mzc3lxtj0vgq2qe
luisqa (Interbloc): cosmos1ze09kc5ackut7wc4pf38lysu45kfz3ms86w3em
tricky (CosmosSpaces): cosmos1a8x3fn37gjnglcr25fsfyr6c5m4ed5euwvae2n
Ertemann (Lavender.Five Nodes): cosmos1xfl6qve3plepgk7wlgxypem5ngntavrnkng3vz
Multisig address - cosmos14r8ff03jkyac2fukjtfrfgaj8ehjlhds5ec2zp