[Proposal #839][ACCEPTED] Fund 2024 Hub development by Informal Systems and Hypha Worker Co-op

[pooch]

@FrancescoSVC

is there really a need to use another 100k ATOM bonus for performance?

I, too, think the bonus is overly generous and “exceeds expectations” is totally nebulous.

Don’t get me wrong, I’m all for funding these teams, but adding the bonus aspect with a subjective condition reeks of wanting the upside potential of ATOM without taking any risk. That’s not fair to those that have spent real money to obtain ATOM.

in3s.com will vote a reluctant “YES” on this proposal. The reluctance is only because of the inclusion of the bonus. (KPIs, as @Rarma suggested, should be used to determine if a bonus is earned.)

[jagachu]

This person analyzed it very well, providing validators and delegators, think about it carefully.

G. 🎒🎺

[Govmos]

We endorse this proposal but wish to highlight the profoundly unfortunate timing of converting ATOMs into USDC at a time when their value is severely depressed. It is imperative that we initiate discussions regarding the establishment of a financial committee to oversee stablecoin reserves and conversions on behalf of the Hub. This step is essential to secure funding for the development costs of core teams. Presenting $4 million worth of ATOMs at a price that may well mark the lowest point in this consolidation cycle is a situation we must avoid in the future. We strongly recommend that the community initiates conversations about creating such a committee as soon as possible.

[GOJWH]

We should consider the timing of execution. Right now, it’s not a good time. In the technical analysis of the chart, if we consider the worst-case scenario and can’t hold the current bottom, we should be prepared for a potential drop of up to 25%. Therefore, we need to save our budget for the next year.

[jacobgadikian]

Saddened by the fact that I’m voting no.

The clear control that the foundation has over Informal shows up even in security incidents, and it’s too much.

FWIW, I strongly support this team, but I do not support Informal Systems handling of recent security matters, which was basically to tell Notional to contact Amulet, who they knew full well wasn’t responding to Notional.

Catch-22’s like that have no place in security.

[Rico041]

Have the following arguments of ICF funding responsibility been addressed?

G. 🎒🎺

[crainbf]

A proposal by Hypha and Informal is up for vote on the Cosmos Hub. They want the community pool to fully fund their work. This is work specifically focused on improving the Cosmos Hub.

Until now, these teams have been funded by the ICF. The proposal asks for $5.7m to fund these teams. This represents 16% of the current community pool of $36m.

Does it make sense to have Hypha and Informal funded through the community pool?

One thing that I like about the proposal is that it would really put the ATOM community in charge of the development of the hub. This is very cool. That is the way for the Cosmos Hub to become a coherent and effective, yet decentralized collective. A new type of organization. If the Cosmos community can figure out how to leverage on-chain governance for this, it would really make the Cosmos Hub into the largest and one of the most effective DAOs in crypto.

For me, the Atom Accelerator DAO is a great example. I’m excited that there is a governance-funded grants program that funded 25 grants with $2.7m. It makes me bullish for the hub that AADAO exists. And it would make me even more bullish if the hub governance is directly funding its core development.

Will it work?
Accountability is a difficult thing. And one needs to create a new mechanism of accountability. In the proposal a system for that is included. Many aspects will be difficult. How can the community influence the roadmap? Who determines what the financial terms and incentives are? How do you check the quality of the work. How are issues resolved?
We are trying to do all these things at the Interchain Foundation and we see how hard this is. This is also going to be a big challenge to do in a decentralized way.

If the Cosmos Hub can figure this out, it would be amazing.

How do I think about this from an Interchain Foundation perspective? Note that these are my own opinions. And that my role is that of a member of the foundation council and also president of the council. I’m one of 3 foundation council members. And everyone has an equal vote.

In my view, the Cosmos Hub is an essential part of the Cosmos ecosystem. It’s the original Cosmos PoS chain. It has pioneered many new technologies and has been finding some impressive early product-market fit with Interchain Security. It also has a passionate community. At times that community that can be full of conflicts and disagreements. And it’s an ecosystem full of vision, new ideas and a drive to bring truly decentralized and sovereign blockchains as tools for the world.

The Cosmos Hub needs to be funded and continue to be developed. And at the ICF we did allocate $3.5m in the 2024 budget for the hub. We think it’s crucial to fund the hub along with other key projects like IBC, CometBFT, CosmosSDK and CosmWasm. The amount we allocated for 2024 for the Cosmos Hub is $3.5m. The Foundation Council is also in favor of deploying those $3.5m via the community pool. Thus giving ATOM governance control over this part of the budget. This money would either be sent directly to the community pool or potentially it would need to be paid directly to the organizations that the hub chooses for development. In either case, it would mean that the effective cost for the hub would be $2.2m since the other $3.5m would be covered by the Interchain Foundation. It would represent 6% of the current community pool.

If this proposal is not accepted, then there are different possibilities. We might fund something similar to the on-chain proposal directly with Informal and Hypha. However, we would only be able to fund a part of the proposal.

I thought it was essential to communicate this to the Cosmos community and those voting in governance. My apologies that we did not do so earlier. I hope this is helpful for everyone to understand how I think about this and how the ICF is committed to contributing budget to it.

[Cosmos_Nanny]

Brian, thanks for sharing.

I have several questions for you and @mpg regarding the allocation and communication of funds ICF apparently earmarked for Cosmos hub development.

• Could you clarify when the decision to allocate $3.5M was made? Additionally, when and how was this allocation communicated to the Informal and Hypha teams, if it was communicated at all?

• According to the ICF’s 2022 report, approximately 16.2% of the total budget of $48.1M, or $7.8M, was allocated to hub work. If we include the $1M allocated to Hypha (assuming their work was exclusively related to hub development as per your report), what is Informal/GmbH take of the remaining $6.8M?

• Was there an overpayment for hub development in 2022?

Questions for @jtremback and @lexa:

• Considering that Hub Dev total outlay was $7.8M and Hypha received $1M, what was Informal’s take for hub dev work in 2022? Why is the current proposal for $5.7M?

• Is it accurate to assume that the $5.7M proposal accounts for the $3.5M guaranteed by the ICF? Does this explain why the cost estimate for the work is 27-36% less than total spend on hub dev in 2022?

• What factors led to this substantial discount if the $5.7M proposal does not anticipate funding from the ICF?

• Could you elaborate on how you intend to manage nearly a third less funding while addressing regular maintenance as well as additional R&D?

• If you were made aware of the $3.5 million earmarked by the ICF, when were you informed of this allocation?

• Furthermore, if this information was available, why was it not disclosed in your proposal?

I would appreciate your prompt responses to these inquiries, given that #839 voting is live.

[ebuchman]

Hi, perhaps I can help a bit here.

The decision on the 3.5M was made last week and has not been communicated to the teams. The hub teams and other core teams have been under significant uncertainty as to the 2024 funding from the ICF. I have been advocating for the ICF to finalize and communicate initial funding decisions since July to give the teams time to prepare and enable more conversation and communication about the upcoming budget cuts for 2024, but unfortunately this has not occurred.

According to the ICF report the Hub funding for 2022 was 7.8M - see the table page 17. It includes work at Informal, IG, Hypha, Iqlusion, Strangelove, Orijtech, a total of ~20 people. Due to issues at IG in 2022, it was decided in late 2022 to move the IG Hub team to Informal starting in 2023. Some of the team left, and Hub funding was streamlined from the ICF to just two orgs, Informal and Hypha. This has significantly simplified the organization of development and reduced costs and greatly improved the overall development velocity around the Hub. Going into 2024, the Informal/Hypha teams have sought to streamline and limit growth to provide a reasonable budget for Hub funding moving forward, but one that still enables the team to deliver on an ambitious roadmap as well as Hub maintenance and support.

Informal did not receive 7.7M for Hub work last year, I’m not sure where you got that number but according to what’s in the ICF report it looks like less than 4M. Informal was growing its Hub team primarily starting in 2022 and took on significant additional responsibility when the Hub team at IG was shut down and transitioned into Informal.

Hope that helps.

[jtremback]

The numbers you’re talking about are from the 2022 transparency report for all of ICF’s funding. We’ve reported our teams’ 2023 numbers in the proposal text: $1.2M for Hypha, $4.5M for Informal’s Hub Team.

We were not aware of the ICF’s Hub development budget for 2024, nor were we aware that ICF was willing to deploy that budget to the community pool or pay it directly to orgs chosen by Hub governance (quoting Brian’s comment here on those being options under consideration). We’re willing to work with the ICF and the liquidation multisig to make this happen.

To be super clear - we came to our $5.7M budget based on the cost of our work in 2023 and our projected roadmap and headcount for 2024. We did not choose $5.7M with the intent to receive additional ICF funding (i.e., planning on receiving a total of $8.9M from two sources) nor with the intent of later revealing that the ICF would subsidize that number.

Our budget is $5.7M for this work regardless of where the funding comes from, same as 2023. This is what we think it will take to do the work we’ve committed to in the proposal and maintain the level of service provided in 2023 (focusing on improvements and R&D for ICS, regular testing, and maintenance). If the ICF can offset $3.5M of this, then the total cost to the community pool when all is said and done will only be $2.2M.

[Cosmos_Nanny]

Hi Ethan,
I appreciate your response.

I calculated $7.7M from the 16.2% allocated towards Cosmos Hub from $48.1M total spent. It is closer to $7.8M as 48,100,000*0,162 = 7.7922. I also recalled you saying IG team working on hub was relocated to Informal, it was unclear to me if this happened during or after the 2022 report. Regardless, the presentation of team composition and breakdown above indicates 14 FTE from Informal (from 8.8 FTE reported in 2022) and 3 from Hypha.

It was an error in communication to say Informal received $7.7M of the total outlay for Cosmos Hub development-- rather I meant to ask what was Informal’s take of the remaining $6.7M seeing that Hypha was paid $1M for their exclusive work on the Cosmos Hub per 2022 report.

I suggest for future reporting, would be more clear to show how much each team’s compensatory take is within a general subcategory rather than just showing how many FTE persons they have working on it.

I’ll edit my message to this end.

Unfortunate that ICF’s funding decision was not communicated to Informal/Hypha teams efficiently.
I don’t think it’s a fair to have community vote on assumed support from the ICF, and the funding request is for $5.7M.

Would you consider putting up a revised proposal to reflect the new funding situation? @jtremback @ebuchman @lexa

[Cosmos_Nanny]

I did see the note in the proposal text about your teams’ 2023 numbers. I was trying to understand them in relation to the 2022 reported numbers on total spend for Cosmos Hub. Working from the numbers provided in the 2022 report, and the numbers provided in Forum and in your proposal – one can infer Informal absorbed the GmbH team working on Cosmos Hub, and Ethan confirmed this in his response.

So it’s natural to wonder how one keeps the preponderant team intact, with similar FTE headcount from 2022 – working on the same (if not more) at less cost.

[stakefish_gov]

There are several concerns with this funding proposal:

1. Development funding should come from ICF, which is their mandate. I checked the interchain treasury, – seems to be healthy.
2. One proposal for two development company is too complicated to manage accountability.
3. This proposal quite large in scope and funding. Any development over $1m generally difficult IMHO.
4. Having liquidation multisig seems complicated. Has anyone thought about tax implication imposed on this group? Is there an entity in place to do it?
5. We are shorting ATOM by selling 70% to USDC. Not good optics. I do understand the projects need it in USD to fund for future payrolls. I think ATOM should be funded directly to the project to be liquidated (and funded more often like quarterly basis).

For all these reasons, it is a difficult proposal to get onboard.

[jtremback]

Thanks for the comment, @stakefish_gov. I’ll address it point by point.

1. The answer to this depends on why you think that development funding should come from the ICF- because they can do a better job allocating funding than the community pool can, or because they have a lot of money? They’ve announced that they plan to put $3.5M in the community pool, and I applaud this. Personally I’d love to see a lot more than that put in the community pool by both the ICF and AiB, and I think that the Hub taking core development into its own hands is a good way show that it’s time to make that happen.
2. We work together very closely, which is why we combined our proposals. That being said, we will present separate quarterly reports to the oversight committee, are graded separately, draw from separate vesting wallets, and our funding can be terminated separately. What are your specific concerns?
3. We have one unified team that takes on many aspects of maintenance, support, production operations, and R&D on the Hub’s core offerings. That is a large scope. In my experience, trying to develop a product with many sub $1m grants ends up being more expensive, as shown by the grant-based Hub program that the ICF had in 2022 and the years before. Each one might be easy to say “yes” to, but you end up with more overhead, a scattershot approach, and less follow-through*
4. In an ideal world, there would be some completely on-chain way to do this with no multisig, but technology (not to mention the liquidity) is not quite there yet. If the Binary Builders DRIP proposal was in production, we would be using that, but right now the multisig is the most practical solution. I’m not aware of any tax implications related to this, and I have participated in this type of operation myself in the past without any tax implications, but each of the multisig participants is responsible for their own tax reporting.
5. Just to get it out of the way, according to CoinMarketCap and CoinGecko, ATOM’s daily trading volume averages at $50-$100M (today it’s around $300M). The $4M liquidation is not likely to make a big impact. Is it bad optically? I don’t think so. We can’t run a professional dev team where we might have to cut people’s salaries in half or fire half the team depending on what the crypto markets are doing any given quarter. Doing a new funding prop every quarter would be a lot of uncertainty and overhead.

Ultimately, this funding prop is a referendum on one thing: Would you like the current Hub team to continue into 2024? If it does not pass, we may have to consider winding down.

*Just a caveat on my opinion of grant programs- They have a very important place. Grant programs can bring in devs, bring in new ideas, and generate excitement in the space. For example, I think that AADAO’s program has been valuable this year, and built excitement and momentum around the Hub. I just don’t think that a grants program can substitute for a focused core team.

[ebuchman]

Thanks, helpful feedback.

We’ve written up a detailed FAQ addressing many of the points that have been raised here and elsewhere. We hope that will clarify most questions, but we’re happy to keep clarifying things here.

At a high level, headcount has reduced overall from 2022 from a combination of increased efficiency in team organization and the associated increase in focus that came from consolidating the Hub team in Informal and Hypha. This proposal is an opportunity to mark and celebrate that as a major milestone in the development of the Hub. Things still aren’t perfect, but they seem to be much better than they’ve ever been.

As for the ICF. Interacting with the ICF can often be difficult and confusing. I’m in a weird role to comment on it for obvious reasons, but even I feel this way. I do think there is still hope (starting with the newly elected FC members, to be announced soon), but suffice it to say this makes ICF a difficult and confusing customer for core dev teams. So this proposal is Informal and Hypha together saying they want to have the Hub directly as a customer, to be more accountable to the Hub, independent of the question of the ICF. This proposal will also allow Informal to be less dependent on the ICF, which reduces overall risk and reduces the issue of my conflict of interest (I’m long overdue to write something about the conflict itself and the relationship between Informal and the ICF, and will aim to do that soon).

There should still be every expectation that the ICF cares about the Hub and supports the Hub in its work and its budget. But this is an opportunity for the Hub to ratify its dev team and start a new chapter of governance, whatever the ICF may do. It’s an opportunity for the Hub to take more responsibility for itself, which everyone seems to want. The ICF will take some time still to work itself out, but it can certainly be expected to continue to contribute and to improve.

It will be especially great to see the ICF contribute by offering some direct support to the community pool (eg. the $3.5M), and by becoming a better customer for dev teams and perhaps even business development functions in the future. The ICF is still allocating a ~$30m budget across the stack in 2024 in ways that will support the Hub both directly and indirectly. Ways which will open funding for new opportunities for growth in the ecosystem. But at the current moment in time, everything would be a lot smoother if the funding for the Hub’s core devs for 2024 (which begins in 2 months) came directly from the Hub in the form of the current proposal that the core team has been working on for months. Even if it means the community pool will only later be reimbursed by the ICF. As written in the FAQ, new proposals at this stage will only serve to drag things out, create more uncertainty for the ecosystem and team, and divert the team from more valuable work on the Hub itself.

Again, summarizing the FAQ: This proposal is a request by Informal and Hypha for the community to recognize and ratify Informal and Hypha’s stabilization of the Hub’s core development and to continue it into 2024, but to shift the ownership from the ICF to the Hub itself. Doing so would improve the political economic organization of the ecosystem. If and when the ICF contributes financially, all the better.

Would be great if we could get everyone on board

[stakefish_gov]

We weren’t involved in the planning of the proposal to understand all the nuance of the issue. But as it stands, ICF has yet to set aside $3.5m to the community and the timing of this proposal makes it difficult. If ICF can speak officially their position on future funding (responsibilities, scope) of the CosmosHub w.r.t to Informal and Hypha, then I think it would give more confidence this proposal. You can very well propose this again at later date.

I think the burden is not just approving the $5.8m, but also the precedence we set on future proposal like this.

[jtremback]

Yea, setting precedent is a big goal with this proposal. Here are some of the precedents that we wanted to set:

• Optimistic revocable funding: We think it’s not good to give grantees big up-front lump sums, but it also isn’t good to make them make new proposals every quarter or whatever. For this reason, we have used vesting accounts to release the funding over the course of the year, and specified procedures to claw back funding if performance goals are not met. These mechanisms will be automated by @Noam’s DRIP mechanism, which we hope to make use of next year.
• Oversight committee: There have been many props with multisigs in the past, but sometimes they haven’t necessarily provided the active oversight they should. We designed the committee with specified meetings, time commitments, and grading standards.
• Per head accounting: We think that per-head accounting is not perfect, but it is the best compromise between a fixed lump sum (which lacks accountability) and completely open books (which hurts flexibility and many companies will not be willing to do). In theory, someone could game this by hiring a lot of unqualified or junior devs, but this should be caught by the oversight process.
• $325k per dev, incl. overhead and profit: This rate includes overhead and profit for the company, and I think it is what is necessary to hire competitively in many markets, and make the prospect of working for the Hub attractive for high quality companies. Once this precedent is set, we should be looking hard at whether a given grantee is worthy of working on the Hub, not whether they are a little bit cheaper.

What kind of statement would you want to see from the ICF? I can’t guarantee anything but I can pass along the message.

[Cosmos_Nanny]

Recognize this is a precedent setting proposal, but afraid we’re at risk of setting the wrong precedent.

We should not vote on a proposal that requests $5.7m when the real funding ask is $2.2m. Asking community to overlook the language and ask amount is inappropriate and makes the request process ripe for future abuse.

It might incentivize projects to initially request larger amounts from the cp, whilst actively seeking additional sources of funding even after the vote period ends. The imperative here is governance assumes that teams that are submitting funding requests have exhausted all tenable sources of funding prior to making their proposal. And have had “finality” with respect to all funding conversations.

Approving a proposal based on updated information that does not amount to more than a mere promise of support from the ICF erodes transparency and trust. In view of the above, I encourage Informal and Hypha teams to submit an updated funding proposal that accurately reflects their financial requirements. Some exposition on applicable terms and conditions for distribution of bonuses also desirable.

Asking voters to ignore the on-chain “ask” in consideration of off-chain developments and agreements is wrong.

[GOJWH]

They set a 25% buffer as a hedge against a decline during the voting period, but they believe that a $4 million liquidation won’t have an impact.

It is recommended that the offer for $2.2 million be made ‘on-chain’.
What started with blockchain must be concluded with blockchain.

To transfer control to the community, return all of ICF’s funds and then delegate responsibility.
Using 20% of community funds for annual maintenance is still premature.

[jacobgadikian]

I really hope that this does not pass in its current form. The moves from ICF ( @crainbf moves were welcomed) leave me feeling like we really don’t know everything that we should.

The fact that we were unable to contact informal except via security provider @Jessysaurusrex is just too risky, since Amulet didn’t get back to us.

So at this time it isn’t possible to support funding Informal because their internal policies are dictated not by what is best for the hub, and maybe not even by what is best for informal, but instead dictated by what is best for the Interchain Foundation – or even – not what is best for ICF either – simply by what is dictated.

If 839 is passed, it is likely that Informal will continue to do what ICF tells it to, as the foundation is more important to the survival of Informal than the hub. I don’t want a hub that does what it is told, but if there’s one thing that can be said of informal, it is that they do what they’re told to by the Interchain Foundation, weather or not that is good for the hub.