Amulet Security Advisory for CometBFT: ASA-2023-002

Research Security
1

ASA-2023-002: Default for BlockParams.MaxBytes consensus parameter may increase block times and affect consensus participation

Component: CometBFT
Criticality: Low
Affected versions: All
Affected users: Validators, Chain Builders + Maintainers

Summary

A default configuration in CometBFT has been found to be large for common use cases, and may affect block times and consensus participation when fully utilized by chain participants. It is advised that chains consider their specific needs for their use case when setting the BlockParams.MaxBytes consensus parameter. Chains are encouraged to evaluate the impact of having proposed blocks with the maximum allowed block size, especially on bandwidth usage and block latency. Additionally, the timeout_propose parameter should be computed using the maximum allowed block size as a reference. This issue does not represent an actively exploitable vulnerability that would result in a direct loss of funds, however it may have a slight impact on block latency depending on a network’s topography.

When setting a large BlockParams.MaxBytes, there are two main implications:

  • Increased bandwidth to propagate a block
  • Increased latency to propagate a block

When combined, this may result in less round participation, and in some cases additional rounds may be required to meet the consensus threshold, which could lead to timeouts depending on the topography of a network and environmental factors. These factors can include the number of validators on a network, geographic distribution, network connectivity (including latency, bandwidth, and reachability), the functionality of the modules implementing the logic for a transaction in your chain, etc. The cost to propagate a 21MB block, the default value for BlockParams.MaxBytes, will be far higher than the cost of propagating a smaller 1MB block. CometBFT recommends tuning this parameter to a smaller limit if full initial-round participation is an important quality for your chain.

Considerations

CometBFT is designed to be configurable by chains, and implements many different configuration variables and parameters to allow chain developers, validators, node operators, and chain participants to customize it best to their use case. A high-performing validator may find it necessary to experiment with tuning local configuration, optimizing network and compute resources, and implementing controls to inhibit spam.

Next Steps for Chains and Validators

To increase awareness of the potential impacts of this default parameter, the CometBFT team has updated the documentation (#1405, v0.34.x, v0.37.x, v0.38.x) for builders and maintainers of chain applications. Additionally, it is recommended that:

  • Chain ecosystems and their maintainers set a BlockParams.MaxBytes configuration appropriate for their use case at the application level; in some cases, fine-tuning BlockParams may require a network upgrade.
  • Chain ecosystems and their maintainers evaluate how gas is used and required on their chain, including gas and fee parameters, no-fee or fee-exempt message policies, and ensure that any custom modules integrate with the gas and fee frameworks. This is especially important for chains that may have implemented custom modules or functionality to allow IBC messages to bypass fees.
  • Chain ecosystems and their maintainers audit all of their currently-set parameters and configurations to ensure that they are appropriate for their use case.
  • All validators develop and implement anti-spam measures on their nodes. Amulet encourages validators to form working groups to collaborate on spam prevention and on tooling that can be implemented by node operators across the Interchain.
  • All validators consider developing and implementing tooling that would allow them to sample incoming transactions to enable them to fine-tune the level of service they would like to provide to be resilient in slowdown scenarios. Amulet also encourages validators to collaborate on tooling that can be implemented by node operators across the Interchain.

The CometBFT team has also revisited all the checks performed by the consensus protocol regarding proposed blocks. This investigation has confirmed that proposed blocks with size exceeding the BlockParams.MaxBytes limit established by the application are not accepted by nodes. The team notwithstanding has decided to introduce additional sanity checks for the size of proposed blocks (#1408), allowing for an early identification and rejection of invalid or oversized blocks. These code changes will be included in the next release of each branch of CometBFT.

As more chains adopt the Interchain Stack for new and cutting-edge use cases, the CometBFT team recommends that all chains regularly evaluate their parameters and configurations to ensure they meet the needs of their ecosystem as their networks mature.

A Github advisory for this issue is available in the CometBFT repository. For more information about CometBFT, see https://docs.cometbft.com.

This issue was raised by Notional labs, who reported it via the vulnerability disclosure channel at security@interchain.io on Friday, September 23, 2023. If you believe you have found a bug in the Interchain Stack or would like to contribute to the program by reporting a bug, please see HackerOne.

Note from Amulet on the Security Advisory Process:

In the interest of timely resolution of this issue for validators and node operators, the Amulet team has chosen to use existing processes and resources for distributing security advisories within the Cosmos and Interchain Ecosystems. Stay tuned as we implement an improved, more robust security advisory distribution system that will provide equitable access to information about security issues in the Interchain Stack.

3 Likes
2

I don’t think that this properly captures what we discussed.

I also wish to formally note that from my personal count, our team requested a call with amulet…

  • 7 times by email

We also invited you to join the slack channel where we were working

  • 8 times by email

Calendly links were sent to you:

  • 4 times by email

I gave you my signal number:

  • Twice by email

And I consistently requested your assistance.

I don’t think that there’s an actual understanding of the issues.

I also requested pre publication review.

Please, call or send a message by signal.

This is what amulet said to me pre-publication:

Hey Jacob,

I wanted to provide you with an update from our team about the issue you raised with us last Friday (tracking number ABC-0002).  Since receiving your report, we've been working with the Interchain, CometBFT, and Hub teams to best diagnose and find a productive path forward.  We intend to publish a security advisory with guidance as soon as possible, likely this week.  For this advisory, I'd like to ask how you and your team would like to be credited in the report (names, social handles) - this is optional, however most reporters request a credit.

Thanks, 
Mo

This was my response to Amulet’s request for information:

Hi Mo,

Notional labs would be a good credit. 

You have not been working with the cosmos hub team as defined by Cosmos hub governance.

Notional labs handles security on the hub, and Amulet has not even been on a call with us. 

The communications I have received so far from amulet do not show an understanding of the issue, and I have repeatedly offered to walk you through the problem. 

Not only do I request credit for our work, I request to see the document that you intend to release.

Regards,
- Jacob

I then followed up with:

Furthermore, you have not been working with the stride team.

This issue happened on the stride blockchain network.

I am diligently attempting to give you a full understanding of the issue.

You ever requested artifacts and stuff, where are those? They're in the slack channel.

This incident spans back two years. 

Please let's try and have a call 

Signal: redacted

@Jessysaurusrex could you please check with Mo on your team and verify that those are my exact words?

Thank you. It’s important to me that my objection to publication without seeing the publication first is known publicly.

This is what Notional is doing at Cosmoverse '23:

PXL_20231001_163542259
PXL_20231001_1635422591920×1446 202 KB

I’m dealing with communications and three highly experienced engineers are working this issue.

2 Likes
3

I think it’s important to acknowledge that the only validated and acknowledged scope of the advisory issued is comet.

Validation of this issue should be very informative. I’m under NDA and can’t provide further detail but Notional is happy to collaborate with anyone seeking to understand this thoroughly. Due to disclosure requirements an NDA may be required, and Notional has offered many.

2 Likes
4

Really important for teams to review and sign the NDAs if they’re keen on collaborating/resolving this issue as soon as possible. This document serves to protect both collaborators and Notional once we disclose key details on the issue at hand.

3 Likes
5

… from the interchain foundation and/or amulet, that is.

The pull request below can be merged to ensure that the foundation’s policies are no longer misleading, though it does not address the concrete reality that there is not a singular cosmos engineering team.

The following pull request could be collaborated on by relevant parties, so that we can mutually improve the cosmos.

6

Thanks for the post on this! I’ve been keeping up with the story before it was posted here, and I think it’s essential to provide security reporters with the chance to review the content of their reports before they are released.

Also what is the cosmos engineering team?

2 Likes
7

Yeah that’s the problem, I don’t actually know

1 Like
10

Perhaps not the most helpful, lets stick to the topic at hand.

11

This is the topic at hand.

Binary Builders, Celestia, and Iqlusion have been very helpful.

Teams funded by the interchain foundation, excluding those above, whole noting that iqlusion has no financial exposure to the foundation, have not been.

It should not have been published.

2 Likes
12

Hey all! I’m also under NDA and can’t say much of anything here, but i find it odd that we can’t make comments on this repository:

My understanding of things is that they aren’t 100% as claimed in this report and there are several teams in the ecosystem that have been misled into thinking this isn’t a problem when it might actually be because of the language in the report.

1 Like
13

double the price of gas

adopt the skip blocksdk

increase maximum block size to 2mb

adjust slashing parameters

validators working group on spam

@ebuchman has been quite clear that he very strongly recommends following amulet recommendations, so according to amulet recommendations, I have set up a validator working group on spam. Please contact me if you would like to join that.

Screenshot_20231015-213448
Screenshot_20231015-213448864×1872 303 KB

chain teams: gas audits

Notional has begun to develop an auditing suite for gas pricing. Please contact me by Twitter DM if you would like consultation on this matter.

We are auditing gas pricing on the cosmos hub as part of proposal 104

1 Like
15

Here’s some short term mitigations that chain teams and validators can use

  • Repository will be actively updated.
  • Don’t blindly change important settings.
  • Chains using x/wasm should not reduce block size below 3mb
  • Validators and chain teams should coordinate on coming up with a unified, per chain set of settings.

No funds are at risk unless they somehow depend on liveness

Anyone can reach out to me directly at twitter.com/gadikian and have a private conversation unless they’re from ICFormal or amulet, in which case it will be strictly necessary to log the conversation in the private repository I’ve been using to report issues.

Amulet has been removed from that repository.

I wish to strongly recommend that no one report security issues to ICFormal until significant changes have been made to security policy.

In the meantime, good contacts include anyone listed in the credits to the README.md file in GitHub - notional-labs/placid: mitigations to p2p-storms.

I condemn the operating procedures that led to the disclosure of this issue without mitigation steps in the strongest possible terms

  • If amulet had even bothered to get on a phone call, we would not be in this situation.
  • If informal had even bothered to get on a phone call, we would not be in this situation.
  • notional labs first reported this issue to ICFormal in 2021 via hacker one. There was an 18 day delay and no payment was made. I believe it was then forgotten about. It was handled by @ebuchman.
  • I am of the opinion that If there was a real investigation into this issue in 2021, it’s possible that UST could have held it’s peg because it was not able to react to external market events because it could not make blocks.
  • If another related issue had been addressed in 2022, the situation would be far better. ICFormal was involved in that reporting process, too.
  • I am of the opinion that informal systems + interchain foundation failed to act in the best interests of the cosmos hub and community
  • I am of the opinion that amulet did not follow sensible guidelines for their role
  • I am of the opinion that conflicts of interest likely driven by @ebuchman shared roles in both orgs coupled with ICF opacity of the type proposal 787 was intended to address reduced the amulet role to merely taking issues from security@interchain.io and delivering them to informal systems by email, and then publishing the findings of the informal systems team
  • if you have security concerns about comet, talk to Celestia since they care.
  • If you have security concerns about SDK, talk to Marko from the SDK team, he is highly responsive, engaged, and opinionated.
  • if you have security concerns about IBC, talk to Susannah Evans from the IBC team
16

Increase block times and affected consensus participation, might be (part of) what killed UST, the largest financial impact issue ever in cosmos.

I am of the opinion that is fully the case.

Show me how there wasn’t an enormous impact from “affected consensus participation” when Luna’s systems relied on routine inputs from an oracle, as well as liveness, which was required for rebalancing.

Then, sit down, take 10 15 or 20 minutes, you can watch that video above, or not, and just think for a minute, what if the chain had been operating correctly?

Then keep in mind that there’s a pattern of issues reported to ICFormal getting exploited as in the quicksilver case.

I don’t know where that information went. That’s why it was a very bad idea to not take care of the issue at the time of its report. Because I know who had it but I don’t know who they gave it to or what was done with it.

So we should just take care of things.

But… We don’t.

1 Like
17

Sane Defaults:

If a default is insane, we should fix it…

1 Like
18

I agree with @jacobgadikian , defaults should be sane.

2 Likes
19
1 Like
20

Caint be helped.

Releasing with the coalition of the don’t like getting rekt

2 Likes
21
1 Like
22

Hi @Jessysaurusrex! Could you edit the top level post to remove Notional’s mention as per Jacob’s request above? Thank you in advance!

23

No it wasn’t. We reported:

Almost immediately after we made our report, we began to experience various forms of retaliation because @buchmanster and the comet crew are on a war for their software fiefdom.

We should really have a because clause here – I want reference to us reporting this, because it is not what we reported. We reported a very serious issue that is composed of many less serious issues. We then immediately began to experience retaliation. What amulets actions really prove is that neither Amulet nor informal actually ran the replication of p2p storms that we at Notional created.

@crainbf dude this stuff has got to stop. If it doesn’t, we’re going to get torn to shreds by actual bad actors, because actual bad actors don’t report bugs. They exploit them and they use the bugs to steal as much as possible from the community. By the time this post was made, amulet was in possession of

  • code
  • video
  • docs

All of which showed something very different from what is described here.

Will be releasing the videos shortly after appropriately censoring them.

Lies must not stand.