-
Current network situation
I had my all 2 public sentries, 1 private sentry, 1 validator in aws Korea region. I started to participate from 7001 and today I reviewed my data communication and this was the data.
data outbound to US 70%, Germany 15%, Japan 9%, rest of the world 6%. I expect more percentage of US dominant for US nodes. As you know, physically US is the most faraway country for Korea(no cable under pacific ocean). So 70% was quite a surprise to me. -
70% ~ 80% US communication. What is a big deal? It is just a testnet.
This result is not because more than 70% of validators are american. Actually I feel they are less than 30%. Then what does it mean? Many of the validators choose US, outside from their own country, on purpose, to gain better connectivity. This effect is one way. If you taste US region connectivity, you have very little incentives to move out from US. More nodes in US, more powerful connectivity in US >> more nodes in US and then infinite loop. This is like a entropy, in a reverse way. Very unnatural and it is centralization.One of the clear evidence I am seeing is that in Korean community, quite a lot of validators want to put their majority of their nodes(sometimes even validator node) in US. I expect no US validator will setup any validator in Korea.(maybe even sentry node!) This shows the reverse entropy situation. -
Why geological centralization is bad? >> I will skip this part hoping everyone in cosmos community understand this.
-
Lets discuss about how to tear apart the US networks to every other continents!!
I am not talking about good willing. I am talking about incentives. Validator will spread out only if there is an incentive to do so. We have good community here, but world does not work only by good will. We need scientific system. In my opinion, about 50% of communication in US is a big success. Now, how can we do that? Sorry but I have no idea…
Lets discuss it!
It’s hard to now how atoms holders will think, but hopefully they view geographic & legal decentralization to be important. I think advertising yourself as a validator that is physically in Korea and owned & operated by an entity subject to Korean law will be an advantage. Figment is physically in Canada, and we are owned and operated under Canadian legal jurisdiction. We think this is a selling feature.
Physical location is one thing, but if you’re hosted on AWS in Korea, you are still dependant on an American company. Hosting in such a way that your infrastructure isn’t subject to American law would set you apart.
This is tangential to your point about geography, but one of the reasons that I think validators should own and operate hardware is that co-lo facilities are much less centralized than cloud infrastructure. There are lots of high quality independent data centres that can host your gear.
yes i plans to put my validator in IDC in Seoul. But recently I felt aws instance has bad performance for its price. I might be thinking to pull out some of my public sentries in IDCs too. It will do better performance and less cost in the long term.
So I would expect Relay nodes to help a lot here.
I’d be interested in working with @bharvest on trying out a Relay node between him and the iqlusion validator
I also concern this very much.
I think that I have the same challenge with you
I expect relay node. and it is very very important for cosmos
i am not following what you are saying here zaki. who is iqlusion? what do you mean relay node between me and him? do you mean a connection between different validator nodes via private sentry?
do we have a clear definition of relay node?
I add this test result here because it is quite relevant subject.
I update some multi-regional sentry setup test results here.
-
I had 2 sentries all in Korea, and yesterday, I add a new sentry in US East.(all aws) US sentry’s peering is great, never fall below 60. I feel quite much better connection to the network. Now I have 3 sentrys connected to my private sentry in Korea and then this one connected to validator in Korea.
-
After 1 day, I checked statistics of my missing blocks. Result was not very intuitive. Although I felt connectivity became much better, missing block thing never goes away at all. I was having about 30 missing in recent 10,000 blocks(0.3%), but after I added US East sentry, it actually slightly increased around 40 blocks(0.4%).
-
Conclusion : Because I already had enough(more than 50) peers in my Korean sentry, the new US East sentry does not affect much on my missing block status. I think this is because, whether sentry is in US or Korea, packet should travel through internet from US to Korea where my validator located. So the result was not quite satisfying.
-
Next move : I should try moving my validator to US region for testing missing block behavior. I will inform this also later this week. Also I need to test “direct connect” between US East sentry and private sentry so that I can get some more stable connection between US network and KR validator.
-
More Thoughts : I felt missing block problem is not about average latency. It depends much on highest latency. We have so many routers and switches to pass through along the journey from US to KR, and some of them is making a instant network congestion several times in an hour. This is just a characteristic of world wide network. So, if my validator is not in US, this missing error thing is quite hard to avoid.
Oh… thank you for your exlpanation!! It clears everything to me. 
Relay nodes work like sentry node but connect in private network. The idea is to let validator nodes only connect to the relay nodes, then let relay nodes connect to public sentries. Relay nodes can then connect to each other in a private network. For example, the relay nodes in Forbole and connect to the relay nodes of b-harvest and connect to BianjieAI. Then we have an Asian regional private network of sentry nodes while not exposing the validator nodes. And then these relay nodes can connect to American based iqlusion, figment and Eueapon based dokia-captial etc. The regional networks can connect together to keep the sync in time, not only rely on the public sentry nodes to keep them in sync.
Thanks, man. we should definitely try this together!
why should we put relay node in a private network？i think it’s better to be public, then it could help anyone who wants to connect the blockchain.@kwunyeung
No. If it is public, it has two disadvantage.
One is security(because it is open to internet), two is performance(because connection between relay node can be done via private network provided by big cloud platform -> better latency and stability).
This kind of private connection should be done in many small groups. Because if a group has too much
connection peers, it can lose both security and performance.
So many small group of relay node connection is the answer!
And in addition in my philosophy, we need some HUMAN TRUST to build such small relay connection group.
Because for each other, we have loosened security attack point. If one of the team’s member decide to
attack another teammate, it will be much easier than attack from outside. So we need trust in small group.
So I think best topology is like a “FaceBook” network.
I want to provide another statistical test result here.
Originally I had 2 Korean sentry , 1 Korean private sentry and then validator.
Missing block percentage was averaging around 0.3%
Adding US East sentry -> totally 3 sentrys running
Missing block percentage was averaging around 0.3% , similar to test 0
Now I removed 1 Korean sentry --> 1 Korean sentry and 1 US East sentry left.
Missing number is similar to 30 / 10000, so no effect by removing 1.
Still, it should make some vulnerablity when network is very bad, but in normal condition, not a
big problem.
And then I removed the last Korean sentry. So my connection is like below.
US East sentry -> private(relay) sentry KR -> validator KR
Now I see some changes.
_missing commits / total commits (missing rate) : _
0/1(0.0%) 0/10(0.0%) 1/50(2.0%) 1/100(1.0%) 3/500(0.6%) 10/1000(1.0%) 19/2000(0.9%) 25/3000(0.8%)
Missing rate increased from 0.3% to 1.0%. About 3 times
It means that you need at least one local public sentry which is very near to validator location.
I will test on KR sentr -> private KR sentry -> KR validator set in final.
Then we can understand better about geological situation and missing behavior.
–> missing commits / total commits (missing rate) :
0/1(0.0%) 0/10(0.0%) 0/50(0.0%) 0/100(0.0%) 2/500(0.4%) 4/1000(0.4%) 8/2000(0.4%) 11/3000(0.3%)
This was much better than test 2, similar to test 1.
conclusion : If you have only one sentry, better put it near validator.
For total, test result for signing stability is below
0 = 1 = 2 = 4 >> 3
Three unintuitive lesson about location and missing blocks:
- If validator is far away from network centre, already have 2 sentrys in your region, put one more sentry
in network centre region does not help much on missing block statistics. - If validator is far away from network centre, put sentry in network centre region doesn’t do much.
- If validator is far away from network centre and you have only one sentry, nearest location by validator is the answer.
–> simple conclusion is that just put every sentry near validator.(test 0)
Thank you for your interests. Hope this discussion goes far beyond so that many non-US validators
get a detailed idea about performance and architecture optimization.
I intend to have my validator node in Singapore with at least one sentry/backup in Hong Kong. Motivation is having my incorporation in HK due to fiscal and regulatory minimal requirements, and being offshore from Western countries. I am interested in joining an Asian relay network.
From Zaki in: Sentry Node Architecture Overview - #33 by gkrizek
I’ve been thinking that validators should consider adding another type of node to their Architecture- Relay Nodes.
Here is the definition of a Relay node. A Relay Node is a full node that only make connections to sentry nodes of other validators that the operator of the Relay Node. it runs with pex disabled. The firewall on the Relay node blocks all connections from ip address other than what is on a white list.
The Relay node operator will white list the ip addresses of other validators sentries on the relay node firewall and the validators will add the Relay node’s id & ip address to persistent peers.
The presence of a small number of relay nodes could help ensure that consensus operates at maximum efficiency.
in summary, we have 4 kinds of nodes so far.
- public sentry node : pex-true & unfixed-peer
- relay node : pex-false & fixed-peer
- private sentry node : pex-false & trusted only peer
- validator
By my intuition, below is one of the best practice.
1 is only for searching peers - a scout. We should gather indepth quality of each peer and send those information to peer analysis server. It is not efficient to use 1 as a communicator because it will be busy spending energy for dynamic peer searching work.
2 is only for communication between the network and validator. Because list of peers in 2 is not dynamically adjusted, we should automatically manage its peer list via peer analysis server. In that way, 2 can be free from being bothered by resource spending for searching new peers, having trouble with unavailable peers, being attacked, etc.
3 is for security of validator - another wall for validator. I think this is optional. If you are confident with defensing 2, you don’t need private sentry node.
4 is signing blocks.
In this architecture, one of the most important role of validator runner is to build smart peer analysis algorithm. This algorithm will decide the best list of peers in real-time. It is a control tower.
In my understanding, this new architecture suggested is in one sentence “customized real-time peer list management strategy”. I think this explanation shows more big picture why we need ‘relay node’. It is a division of labor of public sentry node.
in the definition of zaki’s relay node, strategy might fail if you don’t have smart peer list management because the list of peers in relay node is fixed and it does not have an ability to adjust its peer list. should be managed explicitly.