Hi,
Thanks for taking an interest in our project and taking the time to raise your concern. You’re certainly right that we could/should do a better job of documenting this important design decision.
First thing I would point out is that the most successful live BFT systems in the world today use TCP connections. These are of course Bitcoin and Ethereum. Note these are full production systems that run on an internet scale decentralized p2p network and handle many concerns beyond just active consensus - there’s peer exchange, broadcasting transactions, helping old peers sync, etc. Of course these are probabilistic systems, unlike PBFT and Tendermint. But note that even BFT-Smart, the apparent leading implementation of PBFT-like protocols besides Tendermint, also uses TCP. Correct me if I’m wrong, but I don’t believe Castro’s system has ever been used for or destined for production.
We have very deliberate reasons for using TCP. We want to have connection oriented protocols with our peers because we have to multiplex many concerns over a single connection (transactions, peer exchange, etc.), as Zaki already mentioned. While of course we could use UDP, that would put significant additional burden on the protocol developers that we deemed unnecessary since TCP handles it so well already.
There’s actually another important reason we use TCP. It allows us to implement our consensus protocol more efficiently. Using TCP, we make explicit use of the guarantee that messages we have sent will be received (or else the connection will break). This way, we can reduce the amount of application level messages we need to send over the network - we only send consensus messages to peers that we think they need. As soon as we send a peer a message, we consider it having been received, and thus we don’t need to worry about sending it again. Of course if the connection breaks, the peer starts from scratch, but using TCP gives us very nice guarantees that allows us to implement a more efficient consensus protocol. This is tremendously helpful property to utilize especially in a p2p gossip network as it reduces the number of application-level messages which need to be sent over the network, and of course message overhead is a fundamental bottleneck in such systems. This is how we managed to design a system that can support over 100 validators heterogeneously connected over the public internet. Again, as far as we know, no other PBFT-like system has even come close to such a feat.
This is why Zaki was showing you application-level code. Hopefully this explanation makes what Zaki has already said even clearer.
As for this comment:
The consensus protocol can provide the fault detection and recovery instead of doing it at the transport layer.
This was explicitly about protocols using point-to-point fully connected networks, like how PBFT was designed. As already stated, this is fundamentally insufficient for us.
As for your concern re catching-up. That’s a fine concern, but in the scheme of things, it seems minor compared to the many other things we’re dealing with and hardly worth accusing us of having done no due diligence. We have mechanisms that allow peers to catch up both across heights and across rounds when necessary. If it’s a bit slower because they’re catching up on old TCP messages, it’s a small price to pay for the extra benefits we get. If you can find a serious vulnerability here, we encourage you to submit it to our bug bounty program: HackerOne
As for your continued claims on our lack of due diligence, I would point you to the following:
- We just published a paper with formal proofs of safety and liveness of the Tendermint algorithm. It was presented recently at FLoC to an audience including senior researchers from Microsoft who were quite interested in collaboration. In any case, we would love your feedback/review on the paper if you have the time
[1807.04938] The latest gossip on BFT consensus - Last summer we performed extensive industry standard testing on Tendermint as a distributed BFT database. I encourage you to read the results: Jepsen: Tendermint 0.10.2