In March 2020, a high-severity security vulnerability impacting all versions of Tendermint Core was reported through the Tendermint Bug Bounty program. On April 9, 2020, at 14:00 UTC, Tendermint Core versions 0.33.3, 0.32.10 and 0.31.12 were released with patches for this vulnerability. Versions 0.37.9 and 0.38.3 of the Cosmos SDK and version 2.0.8 of Gaia were released shortly thereafter, with updates to use the latest release of Tendermint Core.
The triage, remediation and communication processes were a collaborative effort by employees of All In Bits GmbH, Informal Inc., and Interchain GmbH.
The Bugs
The vulnerability that was reported demonstrated a pair of flaws in Tendermint logic that, when exploited, could cause a remote panic. These bugs could be exploited by an attacker sending spurious bytes to a target node after the handshake process, causing a memory leak in the target node and ultimately a panic and crash. This vulnerability would have allowed an attacker to carry out a Denial of Service attack against public nodes on Tendermint-powered networks, exploriting one of two vectors.
Denial of Service 1
Tendermint 0.33.2 and earlier did not limit the number of P2P connection requests, and each P2P connection would require a memory allocation. Although this memory was garbage collected once the connection terminated, temporary memory spikes could lead to an out-of-memory exception.
Denial of Service 2
Tendermint 0.33.2 and earlier would sometimes fail to reclaim the activeID of a peer after it was removed in the Mempool reactor. This would only happen when a connection failed before the Peer was created and added to all reactors. RemovePeer would therefore be called before AddPeer, leading to a constantly-increasing activeIDs map and a memory leak. The activeIDs map has a maximum size of 65535 and nodes will panic if their activeID map reaches the maximum. An attacker could create a lot of connection attempts, exploiting Denial of Service 1, and ultimately causing a panic in the target node.
Risk to the Cosmos Hub
We believe that this vulnerability created limited risk to the Cosmos Network. This attack could only be carried out successfully on nodes that have publicly-available IP addresses and which accept connections from the open internet. However, per best practices for running a secure node, most validators who operate the Cosmos Hub choose to actively mitigate against the risk of sentry node attacks by not publicly exposing all of their sentry nodes on the network. Furthermore, as Tendermint does not currently rate limit HTTP endpoint, validators are advised to protect any exposed HTTP endpoints using their own rate-limiting software or other precautions.
Based on our current understanding of the topology of the Cosmos Network and the liveness properties of Tendermint (which require two thirds of the network to connect), it is extremely unlikely that this attack on public sentry nodes could have been leveraged against operators with enough stake to disrupt the network.
Remediation
After receiving the bug report through HackerOne, the Tendermint Core development team was able to validate the vulnerability and identify the two denial-of-service vectors outlined above.
After writing, reviewing, and testing the code comprising the patch, the Tendermint team made a patch available in versions 0.33.3 and 0.32.10.
Vulnerability Coordination
In parallel with the technical remediation work for the issue, the triage team kicked off our internal process for vulnerability notification. In the case of high and critical severity bugs, we are committed to providing pre-notification of security vulnerabilities 24 hours before publishing a public advisory to organizations, projects, and entities dependent on code we have developed.
As part of our round of pre-notification, we proactively reached out to several exchanges, validators and partners, and we posted a message to the Cosmos forum letting the community know that a patch for a high-severity vulnerability would become available at 14:00 UTC on Thursday, April 9. This time was specifically chosen to accommodate a broad range of partners and community members spread across many time zones.
Within a business day of pre-notifying impacted parties that we would be releasing a security update for Tendermint Core, releases were cut for Tendermint, Cosmos SDK and Gaia that fully remediated the issue once service providers patched their software. The reporter was awarded a bounty for the bug reported through HackerOne shortly thereafter.
Retrospective
For security to be successful, it must be a shared responsibility across all stakeholders across the entire Cosmos ecosystem. More than anything else, we are extremely thankful for the ongoing contributions to security being made by our community, and we are thrilled that the work we have done to encourage reporting of vulnerabilities through our HackerOne bug bounty program has been successful to date.
We will continue to post security updates and pre-notifications on Twitter and the Cosmos forum. However, we will also be sending out security updates, including release pre-notifications, through a dedicated Tendermint security mailing list, in order to streamline communication around future security issues. We recommend that all service providers relying on Tendermint Core sign up for this mailing list.
Special thanks to fudongbai for reporting this issue, to Anton Kaliaev for his work on the security advisory, and to Jessy Irwin for her previous work and writing on the Tendermint security process. Thank you also to Alessio Tregalia, Aleksandr Bezobchuk, Adriana Mihai, Ethan Buchman and Marko Baricevic for their work identifying, patching, and communicating this security issue.