Hi most appriciated guidance on what is minimum Ram requirements for a Validator and Node, looked around and did not seem to be noted or i missed.
Most appreciate any guidance.
Shad
1 Like
well, you doesn’t need much. For minimum:
- RAM 4GB
- Ubuntu server 20 LTS
- 1TB SSD (a lot of data needs to be saved by a node)
3 Likes
now add cosmwasm
it goes up rather dramatically
2 Likes
What’s the overhead of cosmwasm? AFAICT it runs with RAM, but I don’t know about production recommendation.
Hey Robert I’m sorry that I missed your question. It varies wildly and that’s one of the reasons that I opposed proposal 69.
So what I can say is that osmosis has pretty much doubled its memory consumption since CW was enabled on it even though there aren’t any contracts to the best of my knowledge currently deployed on osmosis.
I’ve seen this doubling pretty consistently. We also saw it with the recent dig upgrade.
Also and I apologize this gets a little bit hand wavy. I did make a post recently however where I posted some screenshots from our actual servers.
Please note that these are our relay machines and they buy necessity are much higher capacity than what’s actually necessary to validate. We do have a standardized validation spec, and I can share that with you as well:
These machines perform extremely well and so far they have been able to validate every chain that we have thrown at them but I do have to say that it’s quite possible that Juno will soon need 128 GB of RAM. Typically for validator nodes we run in the most restricted configuration possible, so we turn off indexing, we run either from state sync or a snapshot image that has been truncated using state sync, and we turn off every single port other than p2p.
Relayers, as you can see in the spot the CW enabled chain post, are pretty much the opposite configuration of that. They need to have at least one unbonding Of state and we usually go with much more than that. This just keeps things stable and happy and running well. Additionally, research into database performance has been spurred by relaying because relayers make a lot of queries and this gets challenging to the database itself.
1 Like
Thanks. What kind of relayers are you talking about? CW doesn’t need extra relayers.
1 Like
Cw needs differently configured relayers unless doing connection-based relaying. It used to be quite a bit of work but I think that time has passed. Where tokens use the transfer port, CW uses the wasm port.
But I actually wasn’t talking about that.
The restricted configuration is how we run validators.
Our relay machines (we run several for each chain) are configured basically opposite to the validators. To summarize it’s like:
On validators we turn off everything possible.
On relayers (blockchain nodes that we run the actual relay software against), we turn on everything possible.
“Spot the cosmwasm enabled chain” is actual screenshots of the blockchain nodes that Notional uses in relaying
The full relay setup we’re operating is massive, about 30 128gb 16 core machines (ax-101) at hetzner in a docker swarm with minimum 3x replicas per chain because the relayer software does so many queries against it.
Oh, so cosmwasm chain needs special IBC relayers? I thought the relayer are generic and responsible just for transferring IBC packets, without doing any check what’s in there.
Little bit of column A, and a little of column B.
If you are configuring the go relayer or hermes on a per-channel basis, then for cw relaying, you’d specify the wasm port instead of the transfer port.
iirc, at one point full and clear support for cw relaying only existed in the ts-relayer by confio.
When we began to relay wasm (the neta channel on osmosis) the configuration was really complicated.
clients → many connections if desired → many channels if desired
after wasm grew, connection-based relaying was added to at least the go relayer, but I think hermes, also:
2022-02-04T07:01:38.530597Z DEBUG ThreadId(983) packet_cmd{src_chain=osmosis-1 src_port=transfer src_channel=channel-169 dst_chain=juno-1}:generate{id=XuZl6E7XNm}: /ibc.core.channel.v1.MsgRecvPacket from SendPacketEv(SendPacket - h:1-3073290, seq:5, path:channel-169/transfer->channel-47/wasm.juno1v4887y83d6g28puzvt8cl0f3cdhd3y6y9mpysnsp3k8krdm7l6jqgm0rkn, toh:1-1753037, tos:Timestamp(NoTimestamp)))
That above is literally one of the first cosmwasm (I guess the 5th) packets to go between Juno and Osmosis.
tl;dr:
if you configure on a per-channel basis, then you’ll use wasm as the “channel” and the contract as the port.
If you’re configuring on a per-connection basis, then the single connection can accomodate both wasm and transfer channels, in the relayer.
On the two blockchains, the connections always accomodate many channels.
Like all of this stuff, we really couldn’t say for certain weather it would work or not when we were first configuring it.
2nd tl; dr:
Same realyer software, but different configuration.
example:
rpc_timeout = '20s'
account_prefix = 'juno'
key_name = 'juno-relayer'
address_type = { derivation = 'cosmos' }
store_prefix = 'ibc'
default_gas = 125000
max_gas = 3000000
gas_price = { price = 0.0025, denom = 'ujuno' }
gas_adjustment = 0.2
max_msg_num = 30
max_tx_size = 1800000
clock_drift = '15s'
max_block_time = '10s'
trusting_period = '14days'
trust_threshold = { numerator = '1', denominator = '3' }
[chains.packet_filter]
policy = 'allow'
list = [
['wasm.juno1v4887y83d6g28puzvt8cl0f3cdhd3y6y9mpysnsp3k8krdm7l6jqgm0rkn', 'channel-47'], # Osmosis-CW20
]
src:
chain-id: juno-1
client-id: 07-tendermint-0
connection-id: connection-0
channel-id: channel-47
port-id: wasm.juno1v4887y83d6g28puzvt8cl0f3cdhd3y6y9mpysnsp3k8krdm7l6jqgm0rkn
order: unordered
version: ics20-1
dst:
chain-id: osmosis-1
client-id: 07-tendermint-1457
connection-id: connection-1142
channel-id: channel-169
port-id: transfer
order: unordered
version: ics20-1
In this case we can see that there’s a wasm port connected to a transfer port, this is because of a standard called cw-ics20.
In production Notional’s relayers focus on load shedding. So for example we do Osmosis Channel 0- Cosmos Hub Channel 141 first, and on dedicated instances so that we can catch that load. This is because the relayers will query the nodes too much and cause them to stop advancing block height.
Basically the main challenge of relaying is to keep rpc endpoints alive so that hermes or go relayer can make rapid fire queries on those endpoints. As the load increases this gets harder and harder.
relayers are definitely agnostic to packet content, but can look inside of packets, just like your home router.
weather we relay certain channels or ports, however, is entirely up to the relayers. Additionally, relaying is fully voluntary, so if one were to build censored relay software, one could censor.
Does this help?
oh, also, Penumbra seems to be working on encrypted IBC of some sort. I don’t know the details, but find the direction of their work very interesting. https:// I suppose.