On January 8, 2021, the team at Crypto.com reported a medium security vulnerability (“Denial of Service 1”) to the team via security@tendermint.com. On January 12, the Tendermint Core team released v0.34.2, with a patch for Denial of Service 1.
On January 13, further testing revealed an additional vulnerability (“Denial of Service 2”). We determined this vulnerability to be high severity, and followed the process outlined in our security policy. On January 19 at 16:00 UTC, we released Tendermint Core v0.34.3, which included a patch for Denial of Service 2.
Both vulnerabilities impacted Tendermint Core versions 0.34.0 and later, Cosmos SDK v0.40.0, and Gaia v3.0.0. The current Cosmos Hub was unaffected.
The Bugs
The vulnerabilities that were reported demonstrated a pair of flaws in Tendermint Core’s evidence handling code. When exploited by an attacker double-signing transactions, these vulnerabilities could cause a remote panic. This means that an attacker with access to a large number of validators could perform a large denial of service attack; however, most applications are configured to severely punish double-signing such that this kind of attack would be extremely expensive, and therefore infeasible, for any attacker to carry out.
Both vulnerabilities were introduced in Tendermint Core v0.34.0, when we introduced a new way of handling evidence of misbehavior.
Denial of Service 1
In Tendermint Core v0.34.0 and v0.34.1, validators could verify and propose evidence from blocks that were still “in-flight.” In other words, it was possible for validators to verify and propose evidence at heights where blocks haven’t been committed yet.
Blocks that haven’t been committed haven’t been seen by all other validators on the network. So under these conditions, it was possible for nodes to receive evidence at heights that they themselves had not yet seen. When trying to validate this evidence, these nodes would then panic.
Using CVSS rules, we determined that Denial of Service 1 was a medium severity vulnerability.
Denial of Service 2
As part of the new evidence handling flow introduced in Tendermint Core v0.34.0, we added a new Timestamp field to Evidence structs. This timestamp would be calculated using the same algorithm that is used when a block is created and proposed. (This algorithm relies on the timestamp of the last commit from this specific block.)
In Tendermint Core v0.34.0-v0.34.2, the consensus reactor is responsible for forming DuplicateVoteEvidence whenever double signs are observed. However, the current block is still “in flight” when it is being formed by the consensus reactor. It hasn’t been finalized through network consensus yet. This means that different nodes in the network may observe different “last commits” when assigning a timestamp to DuplicateVoteEvidence.
In turn, different nodes could form DuplicateVoteEvidence objects at the same height but with different timestamps. One DuplicateVoteEvidence object (with one timestamp) will then eventually get finalized in the block, but this means that any DuplicateVoteEvidence with a different timestamp is considered invalid. Any node that formed invalid DuplicateVoteEvidence will continue to propose invalid evidence; its peers may see this, and choose to disconnect from this node. This bug means that double signs are DoS vectors in Tendermint Core v0.34.0-v0.34.2.
Using CVSS rules, we determined that Denial of Service 2 was a high severity vulnerability. CVE-2021-21271 has been issued for Denial of Service 2.
Risk to the Cosmos Hub
This pair of vulnerabilities did not pose any risk to the Cosmos Hub, which had not yet upgraded to the 0.34 release series.
Workarounds
There are no workarounds, other than upgrading to a patched version of Tendermint Core.
Remediation
After receiving the initial bug report through security@tendermint.com, the triage team kicked off our internal process for vulnerability assessment and notification. We diagnosed the issue as Denial of Service 1 and determined that it was a medium severity vulnerability. As such, we immediately issued a patch release, v0.34.2, that fixed the issue.
After receiving the second bug report, this time for Denial of Service 2, we reassessed the issue. Once we recognized that this was a high severity issue, we began patching the issue in a private, temporary fork. We spent several days running testnets to verify that we had fixed this issue, and we shared a private release candidate with the team at Crypto.com so that they could also verify the fix. Once we were satisfied with the patch, we provided a pre-notification 24 hours ahead of the release. This release happened publicly on the Cosmos forum, as well as over Twitter and via the Tendermint Core security mailing list.
On January 19 at 16:00 UTC, we released Tendermint Core v0.34.3, which included a patch for Denial of Service 2. Again, we notified our users via the Tendermint security mailing list. Cosmos SDK v0.40.1 was released later that day, and Gaia v3.0.1 was released the following day. Although we typically try to tightly coordinate Tendermint Core, Cosmos SDK and Gaia releases, we were stymied by a GitHub outage around the time of the release.
Further Notes
While testing the fixes for Denial of Service 2, we noticed that evidence from Tendermint Core is not queryable from within the Cosmos SDK. This appears to be a separate issue, and is being tracked from within the Cosmos SDK repo: Query Evidence doesn't return evidences · Issue #8360 · cosmos/cosmos-sdk · GitHub
Vulnerability Coordination
In parallel with the technical remediation work for the issue, the triage team kicked off our internal process for vulnerability notification. In the case of high and critical severity bugs, we are committed to providing pre-notification of security vulnerabilities 24 hours before publishing a public advisory to organizations, projects, and entities dependent on code we have developed.
As part of our round of pre-notification, we proactively reached out to partners that were using impacted versions of Tendermint Core, and we posted a message to the Cosmos forum letting the community know that a patch for a high-severity vulnerability would become available at 16:00 UTC on Tuesday, January 19. This time was specifically chosen to accommodate a broad range of partners and community members spread across many time zones.
Retrospective
For security to be successful, it must be a shared responsibility across all stakeholders across the entire Cosmos ecosystem. More than anything else, we are extremely thankful for the ongoing contributions to security being made by our community, and we are thrilled that the work we have done to encourage reporting of vulnerabilities through our bug bounty program has been successful to date.
We will continue to post security updates and pre-notifications on Twitter and the Cosmos forum. However, we will also be sending out security updates, including release pre-notifications, through a dedicated Tendermint security mailing list, in order to streamline communication around future security issues. We recommend that all service providers relying on Tendermint Core sign up for this mailing list.
Special thanks to the team at Crypto.com for reporting these issues: Tomas Tauber, cyril-crypto, yihuang, and brianatcrypto. Thank you also to Callum Waters and Anton Kaliaev for their work diagnosing and fixing these issues, and to Marko Baricevic for his help verifying these fixes on testnets. Finally, thank you to the members of the Cosmos SDK and Gaia teams for coordinating their updates with us: Aaron Craelius, Adriana Mihai, Alessio Tregalia, Shahan Khatchadourian, and Robert Zaremba.