On June 28, 2020, the Tendermint Core team identified a pair of bugs which, when exploited, could be used to halt networks running Tendermint Core 0.33 or later. On July 2, 2020, at 14:00 UTC, Tendermint Core version 0.33.6 was released with patches for these vulnerabilities. Version 0.38.5 of the Cosmos SDK was released shortly thereafter.
Impacts from these vulnerabilities were initially reported on May 31, 2020, by @njmurarka from Bluzelle Networks; and the Bluzelle team’s ongoing reports were instrumental in identifying and fixing these issues.
Gaia was not impacted by these vulnerabilities.
The triage, remediation and communication processes were a collaborative effort by employees of All In Bits GmbH, Informal Systems Inc., and Interchain GmbH.
The Bugs
Two of the major changes introduced in Tendermint 0.33 were:
- New commit verification logic to support the forthcoming light client
- New commit structure to deduplicate data and dramatically reduce the size of commits
However, each change also introduced a bug to the preexisting verification flows.
Denial of Service
Tendermint versions between 0.33.0 and 0.33.5, inclusive, allowed block proposers to include signatures for the wrong block.
This could happen without malicious intent. A network that had been running for a while and then restarted without changing the chain ID could halt: Correct block proposers would inadvertently include signatures for the wrong block when they saw such signatures, and as a result, these commits would not validate. Subsequently, all proposed blocks would be considered invalid, and the network would halt.
This was what was happening when the Bluzelle team opened their initial report. Although it is a dangerous misconfiguration to reuse chain IDs (since it can result in validator slashing), this situation nonetheless revealed a vulnerability that an attacker could exploit. A malicious validator could sign for incorrect blocks, and the block proposer would blindly include these signatures in the next commit, causing the block to be invalid and consensus to halt forever as no correct validator would vote for the invalid block.
Tendermint 0.33.6 protects against this denial-of-service vector by checking all the signatures are for the block with +2/3 majority before creating a commit.
False Witness
Tendermint versions between 0.33.1 and 0.33.5, inclusive, stopped fully verifying commit signatures during block execution. Instead, they would stop after successfully verifying signatures from +⅔ of the voting power. This verification behavior was consistent with what we expected from light clients, but introduced problems when executed by full nodes.
This meant that proposers could propose blocks with +⅔ valid signatures and -⅓ invalid signatures or other arbitrary signature data. While this doesn’t seem to impact the safety of Tendermint, it means that commits could contain a lot of arbitrary data.
(It was already true that blocks could include invalid data, since they could include invalid transactions filled with arbitrary data; but in that case, the application could determine that they were invalid and punish the proposer. However, applications didn’t–and still don’t–verify commit signatures, trusting Tendermint to do that instead. This means that applications are not able to identify commits containing arbitrary signatures.)
This could impact incentivization logic for any applications that depend on LastCommitInfo sent in BeginBlock, since LastCommitInfo contains information on which validators have signed. As an example, Gaia incentivizes proposers with a bonus for including more than ⅔ of the signatures. But if Gaia had been running an impacted version of Tendermint, a proposer could claim that bonus by including arbitrary signature data for ⅓ of the proposers without waiting for their real signatures. (However, please note that Gaia was not running an impacted version of Tendermint, so this example is strictly hypothetical.)
In Tendermint 0.33.6, full nodes verify all the signatures during block execution. Light clients continue to exit as soon as ⅔ of the signatures are checked and do not check nil votes.
Risk and Impact
We believe that these vulnerabilities had limited impact. As Gaia was not yet running on the Tendermint 0.33 release series, it was not impacted by these vulnerabilities. Consequently, the Cosmos Hub was also unimpacted.
Other networks that were running Tendermint 0.33.0 - 0.33.5 or Cosmos SDK 0.38.0 - 0.38.4 were potentially susceptible to these vulnerabilities. However, we received no reports of halting behavior aside from the report from Bluzelle, so we don’t believe these vulnerabilities were ever exploited.
Remediation
After receiving the initial Bluzelle report, the Tendermint Core development team attempted to reproduce the failure. However, we were not able to reproduce it until we received the output from the /dump_consensus_state RPC endpoint for an impacted node.
From here, by looking at the last_commit field, it became clear that one of the signatures was for a different block than all the others. By reasoning about how this could happen, the problem became obvious: proposers were including any signatures they had seen in commits, not just ones for the correct block. While this has always been true in Tendermint, prior to 0.33, commits included the block ID along with each signature. This meant that it was easy to see which signatures were for the committed block and which were for a different block, and to simply not count the invalid ones. However in 0.33, the block ID data was removed as an optimization, since it’s redundant for all the correct signatures. However, this meant that it was no longer possible to tell that a signature was for the wrong block. It simply appeared invalid.
We want to thank the Bluzelle team for their thorough communication and overall responsiveness throughout this remediation process. Their cooperation was instrumental in finding and fixing this issue.
After writing, reviewing, and testing the code comprising the patch, the Tendermint team made a patch available in 0.33.6.
Vulnerability Disclosure
To disclose this vulnerability, we followed the process outlined in our Security Policy. For critical-, high-, and select medium-severity bugs, we are committed to providing a pre-notification of security vulnerabilities 24 hours before publishing a public advisory.
For our round of pre-notification, we posted a message to the Cosmos forum letting the community know that a patch for a vulnerability would become available at 14:00 UTC on Thursday, July 2. This time was specifically chosen to accommodate a broad range of partners and community members spread across many time zones.
24 hours after the pre-notification, releases were cut for Tendermint and the Cosmos SDK that fully remediated the issue once service providers patched their software, and we updated our post on the Cosmos forum with this information.
Additionally, we sent out notices through the Tendermint Security Mailing List and the @Tendermint_Team and @Cosmos twitter handles for both the pre-notification and the release-notification.
Finally, we requested a CVE ID through GitHub, which serves as our CVE Numbering Authority, and were issued CVE-2020-15091. For more information, please see our security advisory.
Special thanks to @njmurarka and the rest of the Bluzelle team for reporting the Denial of Service issue, to @ebuchman for reporting the False Witness issue and for uncovering the root cause of both issues, and to @melekes and @ebuchman for their patching these issues. Thank you also to @alexanderbez for updating the Cosmos SDK and to @adrianakalpa for managing communications over social media.
As a reminder, we have a bug bounty program and a dedicated security mailing list.