[Proposal #839][ACCEPTED] Fund 2024 Hub development by Informal Systems and Hypha Worker Co-op

Interchain security and IBC are concepts Cosmos pioneered over 5 years ago that are only now reaching adoption and maturity at a larger scale. Many similar projects have emerged in other ecosystems, some with orders of magnitude more funding than what this proposal is requesting. It makes sense for ATOM stakeholders to directly support the development of these technologies instead of having to seek grants from other associated entities. The Hub’s version of Interchain security / shared security gives the most optionality out of all similar architectures, and thus deserves enough funding for this vision to be realized.

Informal / Hypha have shown to be more than capable in successfully launching Replicated Security, with Neutron and Stride becoming consumer chains. In terms of engineering complexity, it was close to the order of difficulty of Ethereum’s Merge.

The only thing I’d like to see in this prop is more of a breakdown of Informal’s headcount (i.e: what are the roles of the 12-16 individuals).

Looking forward to Atomic IBC and megablocks!

10 Likes

Thank you all for the discussion above.

We are very supportive of the teams’ decisions to approach the Cosmos Hub community pool for funding.

It is a priority for us at the ICF to coordinate funding with other communities and teams because we want to foster a self-sustaining ecosystem. We want to foster effective, sovereign, and decentralized communities.

The teams’ decision to go for CP funding helps accelerate this process – enabling us to refine our focus on the core stack and broader ecosystem. We also believe that the Cosmos Hub is a beneficiary of the growth of the broader ecosystem.

We aim to minimize our impact on the Cosmos Hub over the long-term not because we don’t care about the Cosmos Hub but because we deeply care about sovereignty and autonomy.

With that said, we are open and listening to the community’s feedback about their capacity to fund and support components of core development. We are flexible enough to accommodate the decisions made by the community in line with our strategy of supporting where we are needed to foster a self-sustaining and prosperous ecosystem around the Hub.

The Cosmos Hub is very important to us, and we want to see it thrive.

5 Likes

Thank you Lexa for your continued efforts and dedication to the Hub. Makes total sense for the Hub to funds its own development.

However, would agree with Rarma, Noam and others re the need for greater clarity around KPIs, deliverables and how the funds will be spent.

Going forward, I think it would also be good if we could figure out some relatively straightforward way of allowing the community to have some say in the composition of these committees.

2 Likes

Thanks for the detailed write-up @lexa - there’s no doubt about the importance both Hypha and Informal play within the ecosystem.

I’d like to express that I am in favour of having the funding come from the Community Pool - I think this is an important step and would ensure that the accountability is with the Hub rather than another party.

With respect to the above, I am in agreement with Rarma. I think it would be beneficial to have a separate funding request and possibly different KPIs set for the different teams as they would be working on different aspects of the Roadmap.

This is something that I think needs to be defined further before going on-chain.

Take the example of Hypha who mention they will have 2 - 4 individuals working on the Hub. How many of these are software developers vs. other roles? The rates being quoted were for Software Devs.

Taking the mid-points of what was quoted (3 individuals & $165 per hour) this would amount to roughly $400K per individual which might sound fine but we should have more of a breakdown for transparency’s sake.

Overall I am in favour of the proposal idea, just want to see more transparency regarding salary / operational figures and KPIs.

Thanks!

2 Likes

Gets kinda hard to add any constructive criticism after this post. Well done Rarma. Great feedback

3 Likes

Thanks for commenting here, @Rarma. Your response seems to have captured a lot of what everyone else was wondering too. To address your questions, and the questions of others, I’ll address it point by point:

Omnibus proposal

It’s important to have a core team that works on low level research and development of the Hub’s core features (right now this is ICS), while also integrating efforts from other teams. This is how most chains (and most software projects in general) function. Having a core team doing both maintenance and R&D builds expertise and continuity around the core features. If it’s not us, I would want to see another team filling this joint role.

While these items may seem related, they each have different scopes and deliverables, and teams work on them.

I’ll have to disagree on this a bit. Of the areas of focus we list, I think they are all quite related.

Testing & Testnets and Gaia Maintenance are obviously pretty core to the Cosmos Hub. These categories are also where the Hypha and Informal Hub teams work closely together. We could have submitted separate proposals for each team, but since we work so closely together, and our work would be greatly disrupted if one team’s proposal passed and the other didn’t, we decided that logically they should be combined.

A lot of work on ICS is maintenance, fixing issues in the backlog, improving performance, and upgrading to new SDK versions. There are also protocol improvements, as listed above. We anticipate that this ICS work will also encompass implementing and bringing to production variations on ICS developed through our work in the R&D section.

The tasks in the R&D section are less well-defined, since the nature of research is to find the best path forward. However, this section is also tightly focused on creating core infrastructure for the AEZ. Partial Set Security (an evolution of Opt-in Security) is obvious. The high cost of adding each consumer chain with replicated security slows the rate at which the AEZ grows. Partial Set Security provides a way for fewer validators to run each chain. Atomic IBC provides a way for chains in the AEZ to communicate instantly, while also reducing expenses by having them share blocks.

Mesh security and multihop routing are in the R&D section, but they might also be categorized as “go to market”. These are both technologies that are being developed by other teams (with some help from Informal in both cases), but where the Hub has the potential to be a market leader. Our goal with these is to make sure we are on the leading edge of their deployment, and do business development to get other chains to choose the Hub as their first choice for mesh security and multihop routing.

R&D accountability

This doesn’t instill confidence that this is worth funding via the community pool. The proposal should be specific in the items you will build and deliver for the Hub. You’re setting a roadmap of specific items the community would like to see implemented on the Hub, not what you’d like to pick from the list and implement. Another reason this should be funded by the ICF and specific items presented to the community to be implemented on the Hub/AEZ through governance approval.

It would be best to have the community vote on each as separate funding for the year, with their own schedules, deliverables and KPIs to be monitored against.

Researching and developing new technology is never a sure thing, and it’s impossible to predict with 100% certainty what the right path is. This uncertainty comes from the fact that specifications and prototypes are developed, information is gained which influences the course of subsequent research. We expect that either Atomic IBC or Partial Set Security will become the main focus of new feature development during 2024.

Uncertainty on Atomic IBC comes mostly from questions of whether the proposed underlying “megablocks” architecture will impose unacceptable costs on consumer chains in the form of resource contention, or unacceptable constraints on consumer chain’s use of the ABCI++ protocol. We will answer these questions by building a prototype of megablocks during early 2024, analyzing its performance, and developing solutions to enable the full use of ABCI++. Based on this information, we will either move on to hardening megablocks for production and making the modifications necessary to the IBC library, or we will shift focus to other R&D items.

Partial Set Security does not have as much uncertainty associated with it, since it is a relatively straightforward upgrade to ICS. However, the design space for the feature is quite large and there are still unresolved questions about it. We go over some of the questions in the CHIPs discussion phase post linked above.

Some of the concerns around Partial Set Security, ICS, and even Cosmos dPoS in general can be alleviated by proportional slashing, which slashes accidental double signs much more mildly while slashing real double spend attacks much more harshly. In some sense, research on Partial Set Security and proportional slashing could be seen as a more conservative research path as opposed to spending time on Atomic IBC. We’re open to any comments on where we should be focusing.

Requiring new proposals and votes for each phase of research would be a large amount of overhead, and would create a large amount of employment uncertainty for our engineers in a way that annual funding does not. Additionally, there would be perverse incentives at play. After putting a lot of work into specifying an area of research ahead of time, and hiring engineers for it, we would not be incentivized to shift our efforts to a more fruitful area, since that would require ending the grant and returning remaining money. This would result in dead ends being followed and a lack of flexibility. We do not think that this is a good way to do research.

That being said, we absolutely want to be accountable to the community. We are working on a process for Cosmos Hub feature development called CHIPs. We’ve posted it on the forum here to gather community input on the process. This process draws on Ethereum’s EIP process to ensure that features under development have sufficient community review during the process. The process has 9 phases. Phases 1, 2, and 3 will be taking place under the R&D budget in this proposal, while later phases move into the ICS, Gaia Maintenance, and Testing budgets. I’ll look at the first three here:

  1. Discussion Phase: An idea is proposed and discussed on the Cosmos Hub forum. This proposal can be looser or higher level, but introduces and describes an idea, research direction, or feature. Depending on the outcome of this discussion, authors can evaluate if it is a good idea to put more time into the project.

  2. Specification Phase: A design document is created providing detailed information on the proposed feature. The outcome should be a Cosmos Hub Improvement Proposal (CHIP) following the format defined in https://github.com/cosmos/cips. This document should be posted to the Cosmos Hub forum. At this point, the specification is a first draft. It will likely be changed during the Spike, Signaling, and Implementation phases.

  3. Spike Phase: A prototype implementation of the feature is created that enables better estimation of the expected timeline. This implementation is not expected to be fully finished or hardened for production, but is intended to gain additional information about the feasibility and design of the feature.


    (phases 4-10 are related to production implementation, hardening, and deployment, and not under the R&D budget)

Right now, Atomic IBC and Partial Set Security (an idea for a viable version of Opt-in without fraud proofs) are entering the discussion phase. Here are the forum posts for discussion of Atomic IBC Megablocks research and Partial Set Security. They should be in the specification stage in early 2024. We don’t have it ready yet, but we may also put proportional slashing (slash less if only one validator double signs by accident, like Polkadot and Ethereum) into the discussion phase for a 2024 implementation as well.

Once the discussion phase is complete, the finished specifications and plan for prototypes (phase 2) will also be posted to the forum. From this process, the community and oversight committee should be able to judge whether the R&D work we are doing is worth it. If we are not producing useful research, I’m sure we will get many responses to that effect on the forum. From here we can adjust and course-correct if this is necessary. If we are not course-correcting, or the course we are correcting to is crappy, the committee or community can put forward a proposal to cut our funding, possibly leaving funding for maintenance and testnets intact if that part of the mandate is going well.

Licensing

I think research into more restrictive licenses for Hub-funded R&D is a great idea, but we are not in the position to suggest a license, as we are not lawyers. However, we are open to working under any license that the community approves by a signaling proposal.

Headcount

The Informal Hub Team budgets for 14 team members. This includes:

  • A project lead, product owner, and operations manager, who oversee and participate in every part of the team’s work.
  • 5 generalist software engineers, who write most of the code.
  • A protocol designer, who answers complicated questions about protocol correctness
  • A verification engineer who builds and uses automated testing technology like CometMock, Quint, and Model-based testing
  • A finance/economics researcher who answers quantitative questions about the financial impacts of technologies (like liquid staking)
  • A devops engineer who helps Hypha with production operations and works on performance related aspects of the code and infrastructure
  • 1 FTE equivalent for marketing and BD support, to help communicate about the Hub and our work, and find and guide prospective consumer chains.

Our per-head total rate averages around $325K per year, this is how with Informal’s 14 and Hypha’s 3.5 team members, we get to a cost of $5.7M. This total cost includes approximately $225K in salary (and relevant overhead such as employer taxes and benefits), the remainder includes standard overhead margin to pay for operational expenses like legal, HR, finance and other support as well as some profit margin to the companies, to ensure long term sustainable growth. Overall, this corresponds to an hourly rate of around $165, a mid-market rate for consultancies.

We will include details on the number of employees working on the Hub in our quarterly reports. In case we fall below the allocated number of employees during a quarter (due to turnover etc.), we will only withdraw the funding from the multisig needed for the number of employees actually employed.

3 Likes

Schedule of work

Like Jehan says, several of the items in our roadmap are closely integrated. I’ll speak to the maintenance (both Gaia and ICS) and testing components since those are the ones that have a clear schedule.

This is a quarterly schedule for two releases per quarter, equating to a total of eight releases over 2024. At this point, we can’t predict or commit to exact features or core tech version updates for each release – we don’t have the information necessary to make those decisions or estimates yet.

Similarly for ICS support at Hypha, we can’t accurately predict how many chains we might be onboarding (however, we expect this to be in the range of 2-5). For a sovereign-to-consumer transition rehearsal, this is what our schedule looks like:

This work occurs for every rehearsal and must fit into the continuous Gaia release cycle – launch rehearsals, Gaia upgrades, and mainnet upgrades all happen on Wednesdays in approximately the same 15:00 UTC time slot to reduce coordination overhead. We have approximately a two week process for a launch rehearsal.

As part of ICS maintenance and testing, Hypha onboards incoming consumer chains to the Hub’s Replicated Security testnet. Each launch rehearsal has a runway of at least two weeks during which Hypha is on-call and supporting consumer chains both in putting together their Replicated Security chain parameters and approaching the Hub governance process. Hypha works most closely with consumer chain teams during the onboarding process, and then again at mainnet launch when we are on-call alongside the Informal Hub team to troubleshoot, debug, and support their launch on the Cosmos Hub.

Testnet Wednesday events include Gaia upgrades, consumer chain upgrades, consumer chain launch rehearsals, and consumer chain sovereign-to-consumer transition rehearsals. During sovereign-to-consumer rehearsals, Hypha works closely with the consumer chain team and testnet validators over the course of two hours for a coordinated launch.

Our testnet work is measurable less with a deliverable and more with the operational smoothness experienced by validators and the incoming consumer chains onboarding onto the Hub. However, we do publish our Testnet Wednesday reports here on the Hub forum.

KPIs

Testing and maintenance have the overall goal of steady, consistent progress in the development of the Hub. We propose these as metrics for measuring against that goal:

  • Smooth upgrades measured in time per upgrade
  • Testnet operational smoothness measured in time taken every Testnet Wednesday
  • Testnet participation measured in validator participation during upgrades and consumer chain launches
  • Quality of learnings measured in testnet reports

Headcount

Hypha’s current headcount is all focused on the testing and testnet line item. If we work strictly on testing, our 2024 headcount would be 2 - 3 FTE. Over 2024, we plan to be more involved with Gaia maintenance and release integration work, which would push our headcount to the upper end of our estimation (3 - 4) as we take on tasks such as release integration.

On Hypha’s side, all the FTE funded by this proposal are working directly towards our KPIs as eng/ops. We aren’t funding separate marketing or BD out of this budget, though the rate does include general operational expenses.

Ongoing funding and oversight structure

Like Spaydh says, the overall format of core development funding on the Hub is probably going to take many iterations of experimentation. Between RMIT’s recent essays on ICS governance and Noam’s DRIP proposal, there are lots of really interesting directions to investigate.

This is one of the reasons that our proposal is only for a yearlong funding period – by the time we are making a second proposal, research in this area will likely have progressed and we’ll have more options.

Direct responses

Can you please explain how Informals proposal and Notionals proposal differ? Or is this a ‘double spend’ for this area?

- Rarma

Afaik (and Lit or Jacob can chime in here to correct me if need be), Notional’s scoped work is:

  • Monitoring the Hub for vulnerabilities
  • Coordinating with Hub teams to produce a security patch and assist in rollout
  • Advising core teams of issues in upcoming releases for safety or liveness
  • Improvement and development of products like the Hub

Monitoring the Hub is something many teams do, and it’s good for many teams to do this. There’s warranted overlap here. Similarly, improvement and development of products like the Hub has some overlap - it’s an open source repo and many people contribute to it.

However, the other items involve coordinating or advising the core Hub team (that’s us right now). We work with the Notional team on identifying and resolving issues that affect the Hub. For example, Notional has recently been doing security testing using the testnets operated by Hypha and we have been closely in contact throughout this process and potential resolutions for it on the Hub.

4 Likes

Hypha has been doing fantastic work!

I’m really looking forward to wargame Thursdays.

With that said, I believe that the cosmos Hub team at informal is still too much a part of ICFormal. As those who frequent the forum are likely aware, I’m really disappointed with the response from amulet which has mainly been nothing.

Of course in addition to nothing, they also published and miscategorized a security issue. Their publication and miscategorization of this issue has forced myself and our team at notional to work in absolute overdrive.

Informal systems still seems to believe that it is important to go through amulet. I am arguing that it is dangerous to go through amulet, and that there is no need to go through them because notional is working security on the hub. If amulets response had been better, we would gladly work with them just like we would gladly work with hypha and gladly work with informal on security on the cosmos hub.

I do not believe that securing this amount of software is something that can or should be left to a single organization. Basically, security is the responsibility of everybody who touches the software stack. Hypha really seems to get this.

As many people are aware, there are legitimate concerns about conflict of interest with respect to the foundation and informal systems. The hub team at informal seems to act as though it is controlled by the foundation, even in matters of security, I have received responses like well our hands are tied and that’s just the policy we have to adhere to. Unfortunately, the reality is that the policy has left me talking to a brick wall.

Now please know, gentle team members of both hypha and informal, I don’t take making such a suggestion lightly, but what if informal hub team → Hypha?

For me this would dramatically reduce what are currently very serious concerns about efficacy in important matters. When funded by the hub, your hands are not tied. You need to do what’s necessary for the hub, and the foundation clearly wants no part in that, so we need to act like that.

I don’t like needing to separate these concerns like this one bit.

Here’s what I can endorse:

  • hyphas excellent work on testnets, security and onboarding
    • For those concerned with overlap, let’s say that I have been doing primary investigation on an issue which amulet should not have published and hypha has been facilitating that and the collaboration has been great.
  • informal hub teams excellent r&d - ICS shouldn’t work. But it does and that’s extremely impressive. What I mean by it shouldn’t work is that it’s a really challenging piece of technology and I think that the execution there has been fantastic.

what I cannot endorse and must decry

  • An overemphasis on proceduralism at informal, to the detriment of the hub security, and verily all of cosmos.
  • insistence on involving the foundations security contractor when:
    • The foundations security contractor is no more responsive than a brick wall
    • The foundations security contractor releases information against researchers wishes, when researchers are saying hey that would be really bad to release
    • The foundation security contractor is not creative enough to consider that a single issue can exploit multiple layers of the stack
  • “My hands are tied” should never come from a hub funded team wrt the foundation who wishes to cease funding the hub. When the hub has funded us, we work for the hub.

In summary, the foundation shouldn’t have influence on hub funded teams operations.

therefore

I wish to humbly suggest we’re not making a single funding proposal for two separate organizations but instead one organization dedicated to the hub. Now, please keep in mind I realize this is a hell of a bet and a hell of an ask for the people involved and I’m not going to blame anybody if that doesn’t go through. With that said, I do want to try to help to chart and optimal course, then I believe that what I have described is optimal.

if the hub funds it, the foundation doesn’t get to decide it, and that likely means that a new org is needed because informal is hard to distinguish from the foundation and follows foundation policies to its own detriment

licensing

Really cool to hear that @jtremback is open to exploring novel licenses. I’m also not a lawyer, and don’t know if we should cobble something together or consult with a lawyer on what should be cobbled. I think that making ICS(above the commit at which foundation funding ends)+Megablocks+AtomicIBC a hub exclusive through licensure places Gaia in a very competitive position.

one more thing

I frankly find it quite refreshing to hear @jtremback telling the community the direct truth about the non-linear nature of technology research and development. It’s not a sure shot. Jehan is a plain talker and a doer and Gaia is much better off because of his work.

The same is true of the rest of the informal hub team and hypha.

other factors

Additional things that could influence my decision making here would include the separation of the foundation and informal systems. The the funding of informal systems, coupled with Ethan’s role at the foundation have created non-distinct entities.

If this prop means independence for Gaia from ICFormal it has my full and unwavering support.

I don’t support hub funding for ICFormal.

Jehan and Marius’ engineering team is great.

It would be hard to find better stewards and facilitators than Hypha.

5 Likes

Some really great points and feedback mentioned here:

I think a lot of good topics for feedback have been mentioned already and so I wont go into detail. As mentioned by others, it could be better to be more specific about the team members working on the deliverables of the funding and the funding itself to be ‘for XYZ deliverables to be worked on by ABC individuals from Informal and Hypha for EFG period of time’ instead of for ‘the Informal and Hypha teams’

This coupled with a funding amount and people responsible for work and delivery for each single area of work, as mentioned by @Rarma regarding R&D vs Interchain Security vs Gaia Maintenance etc, should allow the community to review each part of the work done by Hypha & Informal more effectively, generally resulting in better accountability and more effective communication between the community & the teams regarding the work conducted.

Seperating the work taken on from this proposals funding and the existing work being conducted by Hypha and Informal outside the scope of this proposal (such as existing work and projects or newly taken on work), from a reporting/accountability perspective, ensures the community can more effectively monito the work conducted and Informal/Hypha can more effectively communicate who by, how and when deliverables have been completed.

Overall, supportive of the Cosmos Hub funding its own development and maintenance costs and these are two great teams to work on Cosmos hub development and maintenance. It’s a large spend, 20% of the community pool for 1 year of development/maintenance work, so good accountability & transparency in communication will be beneficial for both the team and the community.

Look forward to seeing this proposal go on chain.

1 Like

Well, so a corporation really only has one set of books. Which is why I am suggesting that the hub team leave informal.

I don’t like the overhead of that move but informal has done very poorly by the hub lately. But this didn’t really come from the hub team, instead it came from informal and Bucky’s entanglements with the foundation

I agree the points brought up by Liam. I feel like this proposal should be clearer in terms of who is doing what work, and what are the timelines. This allows for better accountability in the long term.

Whilst the rates mentioned are software development rates, is all this work really software development work?

For example, does it really cost $2.5million per year to do testing, testnet coordination, incident response and management? What part of this work also fits within the scope of @jacobgadikian’s Notional responsibilities?

Considering the teams are receiving $325k+ per team member (which seems insane to me), is there really a need to use another 100k ATOM bonus for performance?

2 Likes

more specific about the team members

Jehan wrote an update to this that I’ve also included in the top-level post. On Informal’s side, it details the number and roles included in this proposal. I’ve done the same for Hypha here and in the top-level:

  • 2 general software engineers who write code, design and build tests, and work with other dev teams
  • 1 product manager who manages scope, timeline, communications, and is involved with every part of our team’s work
  • 0.5 devops engineer who manages and orchestrates our fleet of nodes on two public testnets and multiple consumer chains

For a little more context, Hypha’s team is pretty small (3.5 people) and we tend to be mostly heads-down developing and improving systems to ensure that quality code lands on the Hub. Over the last year, we have shepherded drastic improvements to the Hub’s operations and security, as well as the smooth release of new features such as Replicated Security and the Liquid Staking Module. We think we punch above our weight in terms of delivering on our scope of work.

As a note for both our headcounts, we will report on the precise number of FTE employed for each quarter and return any unused funds from the budget.

For example, does it really cost $2.5million per year to do testing, testnet coordination, incident response and management? What part of this work also fits within the scope of Notional responsibilities?

The testing and testnets line item is for $1M. This includes engineering work such as understanding specifications and designing/writing appropriate test cases and tests, creating and maintaining our automated testing framework to enable faster iteration on releases, and working directly with other developers who are using the testnets or working to understand aspects of the Gaia codebase. The things that happen on the testnet are intentionally chaotic and hit new parts of the code, so for critical new features such as key assignment, our team is often the leading expert on the topic.

This work contributes directly to a safe and secure mainnet and provides a platform for testing by folks like Notional (which I spoke to here), consumer chain teams, and any other Hub developers. We think the quality of our work speaks for itself.

The other $1.5M to do the entirety of Gaia maintenance, which includes a regular release cycle, updating dependencies such as the sdk and ibc, integrating version updates from other sources like ICS, working with other teams to scope out new features and how they fit into the release cycle, preparing release candidates and iterating on them in close coordination with the testnets team, and providing support to validators during regular and emergency Hub upgrades.

Incident response and management is part of maintenance, but definitely not all of it.

Considering the teams are receiving $325k+ per team member (which seems insane to me), is there really a need to use another 100k ATOM bonus for performance?

~325k includes operational overhead that is just the cost of running a business. It works out to approximately 225k in employee compensation and we think this is a good standard to set for work on the Hub. We want the best people working here, and that requires a competitive compensation package (including performance-based bonuses) as well as job security.

However, we have adjusted the bonus in the proposal such that no one gets a bonus unless we exceed expectations for the quarter based on Rarma’s feedback. We think this is more in line with what a ‘bonus’ should be – a recognition of exceptional work.

Thank you to both of you for the thoughtful comments :pray:

1 Like

This comment contains details about liquidation and payment mechanisms that will be executed as part of the proposal.

Details of liquidation process

To carry out the goals listed in the 2024 funding proposal, Informal and Hypha will need to hold the majority of the funds in the form of USDC, not ATOM. While we are obviously very aligned with the success of ATOM and will hold 30% of the funds in the form of ATOM to provide added incentivization, the majority of the funding will go to pay employee’s salaries and other fixed costs and for this we need stability.

However, the community pool contains ATOM. So, before the start of our contract in 2024, a significant amount of ATOM will need to be converted into USDC. This USDC will then be put into the vesting accounts described in the funding proposal. Over 2024, this USDC will vest and will be used to pay for our expenses. Keeping 30% of the total budget in ATOM means that 4M worth of ATOM will need to be converted to USDC; there are several ways this could be done.

Potential conversion methods

  1. Fully automated
    • The ATOM amount would be transferred into some mechanism like Timewave, which would automatically and trustlessly convert it to USDC on a Cosmos DEX.
    • This is the most appealing option in principle, but currently there is no mechanism available to do this (let us know if we are wrong).
    • There are also issues with slippage, which we will get into more below.
  2. Multisig with DEX
    • We would name an interim multisig with several trustworthy validators or community members to perform the conversion on a Cosmos DEX.
    • Due to the low volume currently on Cosmos DEXs, this trade would currently incur a high amount of slippage. Most likely, the multisig would need to do many swaps over a period of time to allow enough time for arbitrageurs to backfill the liquidity on whichever DEX we used. This would be demanding for the multisig participants, and the price could move during this time, further complicating matters.
    • It would also be possible to use concentrated liquidity on DEXs that offer it (like Osmosis) to place a liquidity position at a discount below the spot price, and allow it to be filled by arbitrageurs. This might have some advantages over swapping, but it could also involve complications if the price moved. Also the position would have to be closed quickly after it was filled to avoid inadvertently buying the ATOM back.
  3. Multisig with OTC
    • Instead of using a DEX, the interim multisig would use the OTC desk on a centralized exchange. This would allow the entire liquidation to be done in one step, with very little impact on price. $4M can be handled very easily by OTC desks on most centralized exchanges. OTC liquidations by definition do not have an immediate impact on price since they occur off of the order book.
    • The downside of this method is that it would require the interim multisig to put the funds into a centralized exchange account, which is not very decentralized. However, we believe that this is probably the most expedient solution.

Regardless of method, the price impact of liquidating $4M USD worth of ATOM is minimal compared to the 24h trading volume. CoinMarketCap regularly reports a 24 hour trading volume of $50M - 100M USD for ATOM.

Step-by-step liquidation process using OTC deal

  1. Immediately before going on-chain:

    • Use the spot price of ATOM to convert the $5.7M USD budget into a pure ATOM value.
    • Add 25% to cover fluctuation in ATOM price over the 2-week voting period.
    • Add 100k ATOM to this number.
    • This produces the total ATOM amount used in the Community Spend proposal (### ATOM).
  2. If/when proposal passes: Total ATOM amount will be sent from community pool to a 3 of 5 interim multisig comprising:

    • CryptoCrew
    • SimplyStaking
    • Stakecito
    • Stakin
    • Citadel One
  3. The interim multisig will use an OTC deal or swap to liquidate $4M worth of ATOM (using the 25% buffer to cover price fluctuation and fees if needed) into USDC. This $4M represents 70% of the total budget. The remaining 30% will be retained in ATOM.

  4. The interim multisig will send the following transactions to vesting accounts managed by the respective teams. While all accounts are continuously vesting over 2024, the ATOM budget and ATOM portions are divided for ease of transparency and reporting.

    • Informal budget: 3.16M USDC + $1.34M worth of ATOM
    • Informal bonus: 79k ATOM
    • Hypha budget: 840k USDC + $340k worth of ATOM
    • Hypha bonus: 21k ATOM
  5. The interim multisig will return all remaining ATOM buffer to the community pool.

    • If the ATOM price is exactly the same as when the prop went up for voting (2 weeks before this point), then the entire 25% buffer will be returned to the community pool.
    • If the ATOM price went down over the 2 week voting period, then some of the buffer will have been used to cover the difference, and less will be returned.
    • If the ATOM price went up over the 2 week voting period, then more than the 25% buffer will be returned to the community pool – the interim multisig will only sell enough ATOM to cover the amount specified in the funding prop.
  6. The interim multisig will publish a full description of the deal, including documentation of:

    • Trading venue for the OTC deal
    • Price received in the OTC deal (it may be necessary to offer a slight discount from ATOM’s spot price)
    • Any trading or other fees involved

Details of vesting and payment mechanism

Budget vesting

Once the USDC + ATOM budget is in Informal and Hypha’s vesting wallets, it will vest continuously through December 2023 and the 2024 year. The teams will not stake ATOMs which are vesting. Until tokens have vested, they have not been earned and we are not entitled to reward from staking them.

Beginning at the start of January 2024, teams will withdraw the full vested amount of USDC and ATOM on a monthly basis to meet payroll and operational needs for the upcoming month. On a quarterly basis, the teams will report on the burn rate for the quarter (i.e., how many FTE were employed) and return any unused budget to the community pool.

Bonus vesting

The ATOM bonus will vest on the same continuous schedule over 2024. Similarly, teams will not stake vesting ATOMs.

On a quarterly basis:

  1. Tams will present reports to the oversight committee. Reports will also be posted publicly.
  2. The oversight committee will give grades to the teams based on the grading schemes in Appendix C.
  3. If the team receives a grade of ‘Exceeds Expectations’, the quarterly ATOM bonus will be sent to a vesting account with a one year locking period to further incentivize good performance. For example, any bonus earned in Q1 2024 will be locked and inaccessible to teams until Q1 2025
  4. If the team receives a grade of ‘Meets Expectations’ or ‘Needs Improvement’, the entirety of the quarterly ATOM bonus will be immediately sent back to the community pool

I’ve edited this in the top-level post as well but I’ll reiterate here:

Teams will only earn a bonus if they exceed expectations based on the quarterly reports and grades received from the Oversight Committee:

  • “Needs improvement”: 0% of bonus awarded
  • “Meets expectations”: 0% of bonus awarded
  • “Exceeds expectations”: 100% of bonus awarded

Thanks for the proposal Lexa and Jehan, I think you add great value to the Cosmos Hub. Just one comment. The CP funds come from a tax to ATOM stakers and validators. This funding will come from the CP to pay the salaries of several Informal Systems and Hypha employees. However, Hypha and Informal Systems are for profit companies.

If the ATOM stakers and validators are paying the salaries of certain employees of Informal Systems and Hypha, shouldn’t these employees be ‘employed’ by ATOM stakers and validators, for example with some subDAO entity? For example, imagine a person works for company A, then he gets hired by company B and receives a salary from company B. Here what it is suggested is, people work at company A but the ‘company’ B pays the salary for the people at company A. If the Cosmos Hub is paying the salaries, these people should be ‘employed’ by the Cosmos Hub and work exclusively for the Cosmos Hub.
If they are still employed at Informal Systems and Hypha, there are bosses in the companies so they need to answer to them and not to the Cosmos Hub since they are employed there. Also, Informal Systems and Hypha benefit because their overall costs would be lower since many salaries wouldn’t need to be paid, so the earnings would be higher and hence the valuation of these companies which benefits the shareholders of these companies not the ATOM stakers and validators.
I support the proposal, I just find it surprising that Cosmos Hub will pay all these salaries without really ‘hiring’ these people.

2 Likes

I think it is not right to say that the Hub would be paying salaries without actually ‘hiring’ the people themselves. We’re talking in terms of salaries because it’s a convenient way to break down what it means to have a contract worth a certain number of dollars, not because the Hub is directly paying salary directly.

This is an agreement between the Hub and the companies, not the individual employees. The Hub pays the company to deliver the things we’ve outlined in the prop, and the company has its own employment agreements with its employees (including bosses, whose job it is to make sure this work gets delivered). Many people would choose to work for a company over a DAO or subDAO entity because of things like health insurance, benefits, retirement contributions, taxes, etc.

I’d be very interested in seeing a DAO figure out how to offer a competitive employment package that accounts for things like health benefits, vacation time, etc. I think that would be a huge evolution in our ability to do decentralized work. Delphi Labs published content earlier this year about the idea of a legal entity that uses decentralized tech as an integrated part of the entity’s governance and activities. I know some folks in our ecosystem are definitely thinking about this idea and how to keep it moving forward.

2 Likes

This proposal assumes the “community” is positioned to perform accountability efficiently and effectively. It is not.

To date, ATOM Community Pool expenditures and funded recipients have lacked a clear and consistent framework for transparency deliverables and initiatives; overall, even well meaning entities rely on a combination of self-policing and seemingly arbitrary and or partial disclosures.

The hub community has yet to define nominal or normalized standards for “standards” essential to meaningful monitoring, evaluation, and reporting. At present, the “accountability” that exists is entirely self-designed and self regulating AND operates internally within funded teams. The “accountability” is totally dependent on ethical frameworks and internal controls at the design discretion of funded teams.

To be clear, DIY accountability is not accountability.

The community has observed challenges and limitations with internally curated and “ratified” oversight committee functions, as per AADAO operations.

All that said, I recognize the importance of securing hub funds for Informal and Hypha hub teams.

I also conjecture the ICF desires more separation from funding maintenance of a thing that can be interpreted as software use to create digital assets that are securities.

This proposal reinforces to me the importance of, and urgent need for an external and independent oversight hub committee.

And in the absence of such committee, I do recognize the measurable efforts Informal and Hypha teams have taken to differentiate their pre-proposal in articulating deliverables for an accountability framework. You’ve shared alot of ideas on how to make reporting meaningful relating to what’s effectively a service of work agreement with the hub. But we can use for details as to how the financial reporting will be delivered. Some suggestions for your consideration:

  • semiannual and annual financial reports to be delivered with 1) clarity and comprehensibility 2) completeness 3) reliability 4) materiality 5) consistency with respect to presentation and valuation 6) no offsetting of assets or liabilities in income and expenses

  • seminannual report should be inclusive of 1) grant financial report 2) grant cash/crypto transaction report

  • make public all your operational wallets managing hub disbursed ATOM and other funds

  • annual financial reports to include 1) balance sheet 2) income statement 2) cash flow statement

  • add an accountant to your oversight committee

  • Informal Systems and Hypha AG to make necessary disclosures e.g., who owns equity (voting shares and non voting shares) in coop, and how much? Are either entities invested in other projects? If so, in what/who?

1 Like

Add an accountant or a person experienced with preparing financial statements, please.

Some of the roadmap sounds like it can be redundant with R&D initiatives funded by AADAO. What will be done to avoid redundancy with AADAO if they’re renewed in December?

Separately – R&D on ICS insufficiently samples sentiment and experience of operator-stakeholders, namely validators. What will you do to make your R&D more diverse and inclusive of operator/provider perspectives and their ongoing concerns?

Accountability

Really appreciate these two contrasting ideas –

  • urgent need for an external and independent oversight hub committee.
  • the importance of securing hub funds for Informal and Hypha hub teams.

Without any existing accountability structures on the Hub, we’ve done our best to provide the best ‘DIY accountability’ possible. But accountability standards on the Hub are sorely needed, and it would benefit both the community and any entities looking to make proposals.

Financial reporting and disclosure

Our orgs work with accountants, who will be involved in preparing reports. Adding an accountant or another finance-oriented person to the committee is an excellent suggestion, thank you. We’ll look into that.

Our vesting and operational wallets (including any locked vesting wallets for earned bonuses) will absolutely be made public.

We’ll deliver financial reports (balance sheet, income statement, cash flow statements) on Hub-dispersed funds every quarter – this seems like the most sensible cadence to me since there will be monthly payroll withdrawals and quarterly assessments around the performance bonus. Each quarter, we will be able to report on our expenses, return any unused funds, and publicize the relevant transactions. Because it is a proposal for a single year, I think annual/semiannual reports would not provide enough information for the community to make sense of and give feedback on.

As for disclosures, Hypha is incorporated as a non-share capital worker co-operative in Ontario, Canada. Because we don’t have shares, we don’t have shareholders, we don’t pay out dividends, no one owns equity in Hypha. Our corporate governance is 1-member-worker-1-vote and we currently have 7 members and 6 employees/contractors who may become members. One of our members is doing product/engineering work with Timewave, which is a new project building tools for DAO-to-DAO agreements. Other workers’ projects are not directly relevant to our work here but you can read more about those projects here.

3 Likes

Hey Lexa,

To be clear, my use of “DIY accountability” was minus snark. It’s clear to me both teams anticipated these concerns and shared as much re: roadmap, KPIs etc. The effort is both noted and appreciated.

RE:

  • urgent need for an external and independent oversight hub committee.
  • the importance of securing hub funds for Informal and Hypha hub teams.

Contrasting, yes. Although distinct, they don’t necessarily conflict.

Thank you for adding an account/finance person to your committee. An unsolicited suggestion/nomination for you – someone like James Parillo at Figment would be an excellent addition. No idea about his availability or amenability, however.

As for addressing ambiguity in accountability standards – down to get something done on this front.

Any thoughts on creating/authorizing an independent oversight dao via hub governance? Standardization is both an education and an enforcement process. The community can also benefit from a dedicated team to routinely evaluate transparency measures; and streamline community demands for transparency/information requests. Might help reduce FUD tape. Wishful thinking, maybe.

More importantly, think such dao can be a resource hub for increasingly distributed teams lacking in-house experience and orientation with best practices for fiscal management + internal controls. Kinda had an adhoc working group to organize ideas for an oversight committee months ago (alas, distracted by some legal stuff). Happy to pick that up again, and would be great to prompt you for inputs if possible.

For the immediate term (doesn’t have to be a dao) an independent crew w/relevant experience can research, define:

  • minimum reporting requirements,
  • reporting cadences,
  • evidencing requirements,
  • standard templates,
  • and other associated deliverables (practical output measurement pending on funded organization type and funding amount received)

Then share a formal recommendation of reporting requirements for cp funded teams through governance.

1 Like