On October 8, 2021, the team at Provenance reported a high-severity vulnerability in v0.43.x and v0.44.{0,1} of the Cosmos-SDK. On October 12, 2021, a security patch was released in v0.44.2 of the Cosmos-SDK that addresses the issue. Announcements can be found here. Following our vulnerability disclosure policy, we are here releasing more details about the vulnerability, one week after the release.
The Bug
Affected versions of the SDK were vulnerable to a consensus halt due to non-deterministic behaviour in a ValidateBasic method in the x/authz module.
The MsgGrant of the x/authz module contains a Grant field which includes a user-defined expiration time for when the authorization grant expires. In Grant.ValidateBasic(), that time is compared to the node’s local clock time:
See the cosmos-sdk/authorization_grant.go at master · cosmos/cosmos-sdk · GitHub
func (g Grant) ValidateBasic() error {
if g.Expiration.Unix() < time.Now().Unix() {
return sdkerrors.Wrap(ErrInvalidExpirationTime, "Time can't be in the past")
}
Local clock times are subjective and thus non-deterministic. An attacker could craft many Grants, with different but close expiration times (eg. separated by a few seconds), and try to exercise the granted functionality for all of them close to their expiration time. It is likely in such a scenario that some nodes would consider a grant to have expired while others wouldn’t, leading to a consensus halt.
The correct way to utilize the “current time” in a blockchain state machine is to leverage the timestamp included in the Block header, rather than the node’s local time. The block timestamp provides a Byzantine Fault Tolerant clock that is agreed upon by the consensus.
In the case of the authz bug, the check in ValidateBasic served as an additional defensive check that was not strictly necessary as full, correct validation of the grant parameters is done in the keepers. Thus the vulnerability was resolved by removing the check completely for the v0.44.2 release. In a future release, it may be added back in, using the Block timestamp instead of the node’s local time, to restore the extra defensive measure.
Risk
Any chain running an affected version of the SDK with the authz module enabled could be halted by anyone with the ability to send transactions on that chain. Recovery would require applying the patch and rolling back the latest block.
Note that only a select few chains were running an affected version. The Cosmos Hub was not affected.
Remediation
After receiving the initial bug report, the triage team kicked off our internal process for vulnerability assessment and notification. We diagnosed the issue as a consensus halt and determined that it was a high severity vulnerability. As such, we immediately issued a patch release of the Cosmos-SDK, v0.44.2, that fixed the issue. A pre-notification of the vulnerability was announced publicly 24 hours ahead of the release of the v0.44.2 patch. The release was then announced publicly on the Cosmos forum and the Cosmos-SDK mailing list.
Vulnerability Coordination
In parallel with the technical remediation work for the issue, the triage team kicked off our internal process for vulnerability notification. In the case of high and critical severity bugs, we are committed to providing pre-notification of security vulnerabilities 24 hours before publishing a public advisory to organizations, projects, and entities dependent on code we have developed.
As part of our round of pre-notification, we proactively reached out to partners that were using impacted versions of Cosmos-SDK, and we posted a message to the Cosmos forum letting the community know that a patch for a high-severity vulnerability would become available at 14:00 UTC on Tuesday, October 12. This time was specifically chosen to accommodate a broad range of partners and community members spread across many time zones.
Retrospective
For security to be successful, it must be a shared responsibility across all stakeholders across the entire Cosmos ecosystem. More than anything else, we are extremely thankful for the ongoing contributions to security being made by our community, and we are thrilled that the work we have done to encourage reporting of vulnerabilities through our bug bounty program has been successful to date.
We will continue to post security updates and pre-notifications on Cosmos forum and dedicated mailing lists. We recommend that all service providers relying on the Cosmos-SDK sign up for the mailing list.
Special thanks to the team at Provenance for reporting this issue. Thank you also to the team that responded to and resolved this incident: Robert Zaremba, Aaron Craelius, Marko Baricevic, Dev Ojha, Adriana Mihai, and Michelle Leech.