Check_d v4: Decentralized Endpoint Monitoring for Cosmos Hub

Conversation Validation
1

check_d v4: Decentralized Endpoint Monitoring for Cosmos Hub

We have released check_d v4, the latest version of Cumulo’s decentralized monitoring tool for public blockchain endpoints.

check_d continuously tests validator-operated RPC and REST API endpoints from multiple independent geographic locations. The current monitoring network includes:

  • :united_states: United States

  • :european_union: Europe

  • :canada: Canada

Instead of relying on ICMP pings or synthetic availability checks, check_d performs real protocol requests against each endpoint. This makes it possible to measure actual response latency, block height, availability and historical reliability from each region.

For Cosmos Hub mainnet, the live results can already be explored through the following dashboards:

image
image2503×1276 239 KB

What check_d monitors

For every public endpoint, the system collects:

  • Real RPC or REST API response latency

  • Latest reported block height

  • Endpoint availability

  • Regional performance differences

  • Historical reliability over approximately seven days

The resulting dashboards make it easier to inspect how public infrastructure behaves from different parts of the world, rather than from a single monitoring server.

The same monitoring setup is also available for the Cosmos Hub testnet:

image
image2524×1296 235 KB

Use cases

check_d can be useful for:

  • Validators, to compare the availability and performance of their endpoints with other infrastructure providers

  • Delegators, to evaluate whether a validator operates reliable public infrastructure

  • dApp developers and indexers, to identify suitable endpoints based on location, latency and historical availability

  • Infrastructure teams, to detect regional failures or performance degradation

The aggregated monitoring data is also available through public JSON APIs, allowing developers and infrastructure teams to consume the results directly:

These endpoints can be integrated into external dashboards, scripts, monitoring systems or endpoint-selection tools.

What changed in v4

REST checks no longer use Puppeteer

The REST API checker now uses native fetch requests instead of launching Puppeteer.

This significantly reduces checker overhead. In our tests, some reported response times decreased from approximately 1,400 ms to 300 ms, providing a much more accurate representation of endpoint latency.

Historical reliability

Each endpoint now includes a reliability metric representing its percentage of successful checks over approximately the last seven days.

This helps distinguish between an endpoint that is currently online and one that has remained consistently available over time.

Three independent monitoring regions

RPC and API endpoints are currently tested independently from the US, Europe and Canada.

Each region reports its own latency and availability results, making it possible to identify regional performance differences or geographically isolated failures.

Null latency for failed requests

Endpoints that return errors or invalid responses no longer display misleading latency values.

Latency is only recorded when the endpoint provides a valid protocol response.

Aggregator caching

The aggregation service now caches monitoring results, allowing the dashboards and public JSON APIs to return results immediately without waiting for live checker responses.

Open implementation

The implementation and documentation are available in the Cumulo repository:

The repository includes the checkers, aggregation logic and the components used to publish the monitoring results.

We will continue expanding the monitoring network, adding more geographic locations and refining the reliability and performance metrics.

Feedback from validators, developers and other infrastructure operators is welcome.

3 Likes
2

This looks useful, particularly the regional reliability data and real protocol checks.

For client or wallet-side endpoint selection, does the JSON data include—or could it eventually include—details such as rate limits, CORS support, TLS/certificate status, pruning or archival availability, and the time of the latest successful check?

Also, how frequently are the results refreshed, and is there any recommended way to avoid switching endpoints because of short-lived latency changes?

2 Likes
3

Thanks, this kind of feedback is genuinely useful and helps us prioritize what to build next. :folded_hands:

Rate limits, CORS support, TLS certificate status: not supported yet, but we’re considering all three for upcoming versions. On TLS specifically, we can already indicate it partially and indirectly: when a request fails because of a certificate problem (hostname mismatch, expired cert, etc.), the raw error shows up in the detail field, so the information exists but isn’t a clean structured field yet.

Pruning/archival availability: since we run our own curated list of validator resources, we can add this as a proper field there. We’re still expanding that list and plan to review it periodically. That said, it’s worth noting that the large majority of current Cosmos endpoints in the list are pruned already.

Time of the latest successful check: not exposed per endpoint today. We track a reliability percentage smoothed over roughly 7 days of history, but there’s no explicit lastCheckedAt or lastSuccessAt timestamp per entry. This would be a simple addition, since every check cycle is already timestamped internally.

Refresh frequency: regional checkers run every 5 minutes, and the aggregator caches results for another 5 minutes on top of that. So in the worst case, data can be up to about 10 minutes old.

Avoiding switches from short lived latency noise: this isn’t solved on our side yet. reliability is a smoothed 7 day rolling average, so it’s stable and safe to use as your primary health filter. averageLatency, on the other hand, is just a snapshot from that single 5 minute cycle, with no smoothing. If you’re building automatic endpoint selection, we’d suggest filtering first by reliability (for example, only consider endpoints at 95% or above), using latency only as a tiebreaker among already reliable endpoints, and applying your own hysteresis client side (only switch if a candidate is consistently faster across 2 or 3 consecutive polls, not just one). We don’t currently offer a smoothed or rolling latency field that would save you that client side logic, but it’s a fair feature request for a future version.

We’re already planning a next version that implements part of what’s discussed here, so expect movement on some of these points soon. :saluting_face:

2 Likes
4

Thanks for the detailed explanation. The reliability-first approach, with latency used as a tiebreaker and client-side hysteresis, makes sense for wallet-side endpoint selection.

Structured lastSuccessAt data and a smoothed latency field would be particularly useful for distinguishing a genuinely better endpoint from a temporary performance change.

I’ll share these considerations with the Gem Wallet team as part of our ongoing discussions around wallet and endpoint reliability. Looking forward to seeing how the next version develops.

2 Likes