After we got 1 block on gaia-7002,
we started seeing errors like
I[07-26|21:44:34.520] enterPropose: Our turn to propose module=consensus height=2 round=5 proposer=33516EF7B2CF68ECDD0EE4EA5F674C0A8175F095 privValidator="PrivValidator{33516EF7B2CF68ECDD0EE4EA5F674C0A8175F095 LH:2, LR:74, LS:3}"
E[07-26|21:44:34.521] CONSENSUS FAILURE!!! module=consensus err="Unregistered interface types.Evidence" stack="goroutine 1216 [running]:
runtime/debug.Stack(0xc4237d8fe8, 0xd3c340, 0xc42384d850)
/snap/go/2130/src/runtime/debug/stack.go:24 +0xa7
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).receiveRoutine.func1(0xc420118900)
/root/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:558 +0x57
panic(0xd3c340, 0xc42384d850)
/snap/go/2130/src/runtime/panic.go:502 +0x229
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/types.(*Block).MakePartSet(0xc4201084e0, 0x10000, 0x0)
/root/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/types/block.go:118 +0x124
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/state.State.MakeBlock(0xc4207d6820, 0x9, 0x1, 0x0, 0xc42085a420, 0x14, 0x14, 0x1, 0xc42085a460, 0x14, ...)
/root/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/state/state.go:126 +0x22b
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).createProposalBlock(0xc420118900, 0x4, 0x12)
/root/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:932 +0x1b5
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).defaultDecideProposal(0xc420118900, 0x2, 0x5)
/root/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:861 +0x885
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).(github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.defaultDecideProposal)-fm(0x2, 0x5)
/root/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:152 +0x3e
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).enterPropose(0xc420118900, 0x2, 0x5)
/root/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:838 +0x8fd
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).enterNewRound(0xc420118900, 0x2, 0x5)
/root/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:754 +0x9d3
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).addVote(0xc420118900, 0xc421e84780, 0xc4207f2060, 0x28, 0xcc, 0x0, 0x0)
/root/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:1597 +0xaf4
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).tryAddVote(0xc420118900, 0xc421e84780, 0xc4207f2060, 0x28, 0xcc, 0xc4)
/root/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:1459 +0x56
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).handleMsg(0xc420118900, 0xd3e4a0, 0xc42180eb80, 0xc4207f2060, 0x28)
/root/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:628 +0x64f
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).receiveRoutine(0xc420118900, 0x0)
/root/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:580 +0x6d2
created by github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).OnStart
/root/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:295 +0x140
"
What this was cause by the consensus critical bug fix for https://github.com/tendermint/tendermint/issues/2050 but an oversight caused proper serialization to fail.
We have a hotfix to Tendermint in progress and then release v0.23.1 of gaia.
Then everyone will need to update