2.5 million USD (about 350k) atoms between stride and the hub

Conversation
1

I should not need to be writing this.

But nobody else did.

There are 2.5 million USD of atoms (about 350k) stuck between stride and the Hub.

Most likely the problem has something to do with gas configuration that changed when the fee market was added to the hub.

Certainly there’s been more than enough time to inform users of this.

Neither the hub team nor stride bothered to do so.

questions

  • When was this issue known?
  • Was stride informed of changes to gas?
  • Why has there been no information provided to users?
1 Like
2

Hey Jacob! Thanks for raising this. I mentioned to you this morning that I needed to chat with our eng team to get a bit more context. I have that context now and we’re working on a broader post with all of the details, but here’s some background:

and a TLDR for non-technical folks:

We definitely weren’t trying to hide anything, in this case we’ve been in touch with impacted users directly from the start, which is how we usually handle issues where some and not all users are impacted (in this case most unbondings were able to process normally during this time). We’re working up a longer blog post / postmortem for the issue now, and we’ll use twitter / our blog to convey issues like this in the future to ensure people are better informed.

2 Likes
3

But impacted users were every staked atom user, if they tried to unbond.

4

Not quite. It only impacted certain undelegation batches that involved an excess of LSM liquid stakes. Here’s the explanation from Vishal’s tweet:

Normally, this wouldn’t be an issue, but Stride accepts LSM delegations on the Cosmos Hub. The way this works today is Stride accepts the delegations, then rebalances+undelegates afterwards towards Stride’s target delegations (subject to standard Cosmos SDK x/staking restrictions). Depending on the LSM delegations that Stride has, unbondings can stochastically fail. In particular, sometimes Stride will try to unbond from 35-40+ validators, which consumes > 30M gas

Most undelegations processed as normal. The larger impacted users reached out to us and we’ve been talking with them directly.

1 Like
5

And the smaller impacted users?

What was done to warn them?

1000005886
10000058861008×2244 220 KB

Like this guy

Here is a good example of a footgun warning, from Penumbra

See how it leads people away from danger?

I think that is a good approach.

Because there was absolutely legitimate danger.

People really do not like to be suddenly unable to access their assets.

And that’s really very sensible.

Yet the dashboard shows: “all good yo”

That’s a huge footgun left for users. It does not need to be there.

6

Anyone who was impacted and opened up a ticket has gotten help from our support team (including, AFAIK, the user that made this tweet)!

I definitely can appreciate the point that we should have posted this on twitter or the status page, and we’ll be adjusting the way we handle these comms to do so moving forward. Re: the status page, this is due to a missing flag that we’ll be adding that should reflect this issue in the future.

7

Informal was aware of this problem, and did not inform the user community?

8

I can’t really speak for informal on this, I just have context on the actions we’ve taken at Stride to help get these edge-case undelegations “unstuck” (so to speak).

9

Do you think that stride has been given stable platform services ?

10

How about that peg?

We have invested so heavily in it

But now, statom isn’t redeemable in some cases.

And it has been that way for a month.

But the community only learned of this today. Except some knew much earlier.

Why was I told today that stride is not broken on the hub?

I don’t get it, the present situation is fully incompatible with that claim.

Since there are users who are aware of this, and users who aren’t, is it possible the information was bandied about in a privileged manner that allowed some access to the exits faster than others?

That… Seems certain to me.

@RoboMcGobo are you saying that this condition existed, and Stride did not speak to informal?

…well that would be quite bad, deeply irresponsible even.

Otherwise, I think it is essential for atom holders to know when informal knew that their upgrade had broken redeemability.

@jtremback so… When did you learn that 330k statom were stuck? What did you do to address it? Did stride inform you?

Why can’t I get answers to basic questions?

I am not a security researcher.

…at least that’s a really popular claim at informal.

Why did I need to put a bow on this for you?

Why did reports of this come directly from affected community members?

the treatment of this incident, rather than the incident itself, calls the hubs economic security into question

It is also important to note that statom has become important to defi in the wider cosmos ecosystem.

Information concerning it shouldn’t be given by some not security researcher rando who just happens to reply to a guy on Twitter.

11

So, let’s see what’s happened:

I spend a full day finding out what’s up

1000005896

Total silence from icformal and stride on the issues I think matter.

Status page be like:

1000005898
10000058981008×2244 220 KB

That doesn’t make sense.

Why does no one care that users are losing their shirts?

I want to know:

  • Did informal ask stride not to say anything? (there’s a lot of precedent here)
  • When did stride speak to informal? (If they did not, well, yeah that would be strides fault big time)
  • What has informal done to help?
  • Why not param change gov prop on the hub, weeks ago? (I am NOT blaming stride here. Issue is solvable on the hub side, with ease, without guesswork)

I think that these are very reasonable questions to be asking at this time.

1 Like
12

Further oddity.

Just learned that stride is not sure if they told informal anything at all.

For a month.

While user funds are stuck.

1000005901
10000059011007×1120 160 KB

1000005900
10000059001007×793 165 KB

that doesn’t make sense and this most certainly isn’t misinformation

13

Keep up the work, Jacobgadikian. Your work is extremely appreciated by, if not all, other communities of the Cosmos Hub.

1 Like
14

Just wanted to post this so that everybody is aware, yesterday the stride status page did actually show that the funds were stuck.

It now shows all systems operational. If anyone has funds stuck, you should probably say so here, but as far as I know, stride is now working again.

Yesterday:

1000005938
10000059381008×2244 226 KB

Today:

1000005947
10000059471008×2244 220 KB

When there are issues like these, it is important that the community is made aware of them, because their assets can get stuck.