PROPOSAL #90 [REJECTED] Community Health research, analytics and benchmarking

Hey, I’m Danielo, part of RnDAO. I’d love to hear your feedback on this proposal to champion healthier Web3 communities and provide insights to advance the Cosmos Hub and Cosmos ecosystem.

Simple Summary

If Web3 is all about communities, how do we know if we’re heading in the right direction?

We’ve been advancing a research project to:

  • develop a framework for Community Health with actionable metrics
  • create an open source data collection tool
  • implement the tool in Cosmos Hub and others
  • analyse the interactions and perceptions data to validate the framework and provide insights to advance the Cosmos community.

We’re seeking $30,000 or equivalent as grant match-funding to complete the project.

For clarity, the output of this grant is a community health assessment (including conceptual framework, designing KPIs, building the data collection tool, etc). And a range of more specific, actionable insights on how to improve community health. We can’t say what the specific insights are before doing the research, though.

About RnDAO

We’re an R&D DAO with a mission to empower humane collaboration. Having already delivered research projects on sub-DAOs, DAO Conceptual Foundations, Decentralised Leadership, DAO Compensation and more: http:// rndao.mirror.xyz

Abstract

Like many others, Cosmos depends on the health of its community and its vibez—yet understanding and measuring these factors (what we call Community Health) is challenging.

Today, Web3 communities are left to rely on infrequent transactions data (on-chain records), while data from the significantly more frequent social interactions is limited to basic indicators of Discord or Discourse, Web2-oriented bots like Statsbot that give simplistic measurements, or a patchwork of “homemade” surveys to fill in the gaps. These solutions are ill-suited for DAO communities, surveys are time-consuming for community managers and contributors to use, and the results of these approaches, hampered by poor indicators and/or poor sampling, are unreliable.

Perhaps even direr, the lack of real-time analytics leaves DAO community leaders and members without established baselines to measure against to understand the impact of community-focused initiatives, identify best practices, monitor shocks to the system, or rapidly gauge the effects of system-wide changes (such as market crashes, protocol migrations, etc.).

Specification

For the conceptual framework, we have assembled a team including two PhDs in network science and an organisation designer with significant DAO and community building experience (myself) to bridge both theory and practice. We’ve reviewed over 50 papers on the topics of Community, Social Network Analysis, Resilience, Trust, Engagement, and more. And are synthesising all of these findings to define the key indicators that have high validity and high predictive capacity for community health, while also taking a holistic perspective that accounts for member’s wellbeing and planning to collect data across communities to offer an ecosystem health scrore too.

For the data collection tool, we’re going beyond traditional surveys.

This research proposal focuses on the use of two critical techniques as a starting point:

Organizational Network Analysis (ONA) is a structured way to visualize how communications, information, and value creation occur through an organization based on interaction graphs. ONA has been shown to provide a wide range of insights to improve contributor retention, avoid member burnout, predict team performance and community resilience, identify key contributors, enable decentralization, and improve coordination. Although relatively new, ONA is gaining in popularity over traditional survey tools.

Pulse Surveys are frequent and automated micro-surveys that provide qualitative and quantitative insights. In traditional organizations, they have been shown to increase employee response rate and employee engagement with related initiatives. They’re also used as a tool for culture design and implementing culture change. Lastly, Pulse Surveys significantly reduce admin work for community managers and related roles. Importantly, these surveys offer an opt-out option from future prompts and are also framed to leverage coaching techniques to ‘offer a prompt for reflection’, thus offering value to participants as a recentering exercise on the value of community.

The data is anonymised and collected in a central repository for this first phase (we’re exploring decentralised hosting) and managed by a team having received ethics training and at risk of losing their credentials should it be misused.

Rationale

The techniques used (ONA based on communication and Pulse Surveys) provide maximum insights on Community Health compared with exclusively on-chain data analysis. And provide minimum disruption for community members and minimal admin compared to long-form surveys and user interviews. Crucially, the techniques selected and the usage of Discord messages increase participation by those less likely to respond to long surveys e.g. those less engaged and thus likely with the most valuable feedback to give.

Although our initial focus is Community Health metrics, the proposed approach sets the foundation for further applications. The combination of ONA and pulse surveys offers unprecedented actionable insights in real-time. Some of the potential applications and insights for Cosmos Hub and Cosmos-Chains-based DAOs are:

  • Generate baseline metrics for Community Health / vibez to quantify and better understand the impact that a specific event is having on a community and/or sub-groups within the community
  • Predict which contributors are likely to leave the DAO and take preventive action (without breaching privacy)
  • Build funnels to track member onboarding and identify areas for improvement
  • Identify measurements of decentralization to serve as KPIs or Insights metrics
  • Monitor specific topics like contributor wellbeing, alignment, community experience, etc. in near real-time
  • Attract talent and investment with objective Community Health metrics instead of proxy metrics like member count or proposal count, or financial metrics such as TVL
  • Help new contributors find context-rich mentors outside of the existing pool of well-known but time-poor candidates

In addition to the initial research on Community Health, the potential applications mentioned above (and others to be found) can enable more effective and targeted efforts to build healthier DAO communities.

This research also helps reduce the tooling gap in DAOs compared to the employee and stakeholder experience at traditional corporations.

Why our Team

Team Leads

Katerinabc

Ph.D. in Team Dynamics using Social Network Analysis, Teaching Collaboration, and Organizational Performance at Northwestern University (since 2016).

Co-organized Learning in Networks sessions at the International Conference of Social Network Analysis (2018 - 2020), and previously advised a people analytics company on social network metrics.

Thegadget.eth

Software Engineer. Previously, Product Manager at Neolyze (Business Intelligence Dashboard for Instagram).

Danielo

Previously, Head of Governance at Aragon, 8 years experience in Organization Design consulting (clients include Google, BCG, Daymler, The UN, and multiple startups), and visiting lecturer at Oxford University.

Note: Other RnDAO members will participate throughout the process in user research, literature review, tool development, and workshop facilitation.

Team Advisor

Sam

Previously, Head of Technical Research at Aragon. Previously, Lead Developer of the official JavaScript API for the Ethereum blockchain.

Budget & Timeline

Budget

$30,000 in USDC (equivalent to 3,165 ATOM using lower threshold for today dec 12)*

DAO Health Conceptual Framework and translation into actionable metrics: $8,000

Data Collection Tool Development (discord bot for interactions data collection, config front-end, and results dashboard): $17,000

Data Analysis, Insights Report, and Community Workshop: $5,000

  • the requested budget forms part of a total $90,000 target. Total project costs will be split across multiple, participating DAOs (grants already received from Aragon and Aave).

Estimated Timeline taking into account a 1 week’s delay due to the end of year holidays

  1. Kickoff (grant approval)
  2. update of the conceptual framework and publication - estimated week 2-3 from kickoff
  3. implementation of the tool in Cosmos’s discord - estimated week 3-4 from kickoff
  4. delivery of the Community Health report & workshop - estimated week 4-7 from kickoff

FAQ

What kind of insights can this proposal deliver?
Although part of the work is precisely figuring out what metrics to use and which to discard, we’ve already spent significant time developing a framework to define community and community health and will use this as our starting point, further refining with feedback from the community.

How will the insights add value to Cosmos?
A weak community will fall apart in the face of difficulties while a strong community will come together stronger. The insights enable the Cosmos community to understand which areas to target with initiatives so to improve community health. Instead of shooting in the dark, the insights provide strategic focus and clarity.

Will this proposal have a continuation?
This is very much the start of what could become an ongoing workstream of research and action. We’re hoping the community is satisfied with this pilot so that we can then make subsequent proposals to both deepen the research further and action insights to make the community stronger.

What reassurance can you provide for accountability?
We’re part of a DAO that’s been around for over a year, and both the DAO and we, the leads of this proposal, have invested a hundred plus hours building up our reputation in the space with the names and social profiles we have disclosed. The reputational damage would cost multiple times the 30k being requested (not even counting the lost opportunity of building trust with the Cosmos community so we may make subsequent proposals). Of course, there’s an uncertainty component, as with any R&D initiative, but you can count on us to do everything in our power to make this a success.

2 Likes

Interesting approach.

Are you mainly focussed on DAO’s? Since not everything is currently created as a DAO.
Or what do you see as a DAO? Since that starting point makes quite the difference regarding your targetted audience.

Furthermore, what happens after week 16? We will have a set of recommendations if I read it correctly, but who will do the required follow-up? Only having recommendations is not worth money imo, we need implementation to make the spending of money be of added value

1 Like

Thank you for your question!
The focus is primarily on DAOs but the approach can also evaluate other types of Web3 Communities (the limiting factors are the platforms in which they operate as we’re starting with Discord and plan to expand to other data sources, although that will take a bit of time).

After week 16 the project continues. We don’t know yet what the recommendations coming out of the workshop will be, so it’s hard to discuss execution of any subsequent initiatives. But we do hope to continue supporting the Cosmos ecosystem and will likely make subsequent proposals or invite others (as appropriate) to do so.

Does that work for you?

Please let us know if you have any more questions/doubts/concerns/feedback :slight_smile:

1 Like

Thanks for your answer!

I was just looking to your budget and timeline. If I understand correctly you are targetting a $90k budget of which $60k has already been approved. The remaining $30k is requested from the Cosmos community.
The split in expected spending is a nice $30k. Where is the other $60k spend on?

Because with 16 weeks timeline and $90k funding this means a spending of more than $5k per week. Which is a lot imo for research.

I have been intending to follow up on this post for sometime, but have been delayed due to the flurry of governance activity recently. I suspect this is the case for many others as well.

Thank you @danielo for your proposal. I have attended three RnDAO events and followed along with the work produced by various RnDAO team members. I believe that working with the RnDAO team would be highly beneficial to the Cosmos Hub.

However, I would not advocate for only the creation of the tool. Cosmos Hub governance is currently very young, and lacks the basic foundation that may be present in other DAOs. Likewise, Cosmos has a potentially unique governance model and structure that is related to its underlying principles and values. As such, alongside the creation of the Community Health tooling, I would also advocate for an ongoing RnDAO consultancy effort for the Cosmos Hub. Of course, any such work would need to be coordinated with other entities that may be already serving or seek to serve a similar role. This is indirectly mentioned here:

My overall opinion is that establishing governance foundations are more relevant to the Hub than the development of governance tooling. I also think that RnDAO could provide worthwhile insight to help establish the aforementioned foundation. But, before doing so, we would need coordination across multiple stakeholder groups and clear ownership of the project from the Hub DAO, but I, personally, am not sure where to begin such a coordination effort. There are others like @hxrts @uditvira @Youssef that may be able to provide additional context or thoughts here, however I assume that they are also quite tied up with the ATOM 2.0 whitepaper and Charter Working Draft.

Thanks again @danielo and looking forward to continuing this conversation.

2 Likes

Thanks for asking. So we’re actually spending more than that (our total expense now is projected closer to $150k, most of the shortfall is being covered by the team who deeply believes in this idea). As such, we’re now asking a few more organizations to join Aave, Aragon, MetCartel, and hopefully Cosmos too in helping us advance this project.

The reason for the budget expansion is that now having completed the PoC, we’re targeting the development of a fully functioning tool and will continue to deepen the analysis. This enables us to provide the health assessment not only as a one-off but as the real-time analytics we envision can help community members co-shape their communities on the flight with a deep understanding of the social dynamics taking place. We could also provide insights to not only Cosmos but also other Cosmos-based communities (currently impossible with the PoC given the number of manual tasks required).

We can show you the current prototype over a call (or happy to present it more broadly over an AMA sort of session), as it’s a bit hard to convey the full scope without showing the insights we’re getting. Feel free to message me on what’s suitable and happy to find a time that works :slight_smile:

Thank you for the suggestions!

It’s often tricky to figure out where to start these projects. A few years of consulting experience have taught me the hard way, that’s often best to start with something small and perhaps peripheral that allows us to learn more about how to navigate the ecosystem and then provide the next bit of help.

The community health tool that’s taking shape could provide us that foundation as it allows us to gain an overview of what’s going on, identify key areas for targetted support, and most importantly, know how the initiatives that you/we undertake are impacting the community.

That being said, I’m very open to having a call and exploring how we might support further.

We’re also advancing a separate research project focused on Decision Research in partnership with Mission Publique (a large network of democracy researchers), and they have applied for some funding from Cosmos I understand, so that would allow us to help shape the vision for how governance could evolve over the medium to long term (the short term requiring a faster approach).

In conclusion, I remain at your disposal should you desire to explore how we might support the development of Cosmos’ governance, and encourage us to move forward with the Community Health project as a stepping stone to cultivate the relationship between RnDAO and Cosmos and provide you with value in the short term.

Sounds interesting!

Bold question, can you create a screen recording with spoken text first? Then we can also prepare a call :slight_smile:

I am sure there are some pretty nice community calls which we can join. Citizen Cosmos or Seedizens might be a nice one.

Here’s a quick overview

From that base, we can then do all kind of things based on the network patterns

1 Like

In light of recent events, after seeing the product demo, and given that foundational tools are important for building strong structures, I would fully support this proposal.

RnDAO’s work is consistently high-quality, and it would be helpful for the community to take a more methodical approach to measuring and improving community health. Additionally, the community health tool could benefit the Cosmos ecosystem as a whole, as you mentioned:

In line with what @LeonoorsCryptoman said, perhaps we can arrange a call to gather others’ feedback, and start moving toward an on-chain proposal.

Here are proposed steps forward with that in mind:

  • arrange a preliminary call with interested stakeholders (by December 1st)
    • goal: determine a proposal pipeline (how can we get more eyes on the proposal, who do we need to coordinate with, when should we put the proposal on-chain, etc.)
  • input a last-call date for the proposal draft and share the proposal across relevant platforms, such as twitter, telegram
  • host twitter spaces event for the community to learn about the proposal
  • put proposal on-chain

If anyone else is interested in participating in the preliminary call, please leave a comment below and we can work to arrange a call for the week of December 5th. Additionally, I’ve created an open telegram group (you can join by clicking here) to discuss the details of the above proposed path forward, such as call-time planning, twitter space coordination, and other items that are easier-communicated off-forum.

What are all of your thoughts (@danielo, @LeonoorsCryptoman, @lexa, @catdotfish) on that path forward and method of collaborating?

1 Like

This sounds great!

Thank you for your support and leadership helping us move it forward

1 Like

As a quick update here: a telegram group has been formed to iron out the logistical details of the proposal process (outlined above), those who are interested can join here.

I received some questions related to data privacy and sensitive data that’ll share here as they’re of public interest:

Is it GDPR compliant?

  • The short answer is yes. We’re working with advisors from LexDAO to figure this out. GDPR is a big complex thing (the regulation was heavily influenced by big tech so they’d have a moat). It will take us some time to achieve certification status but we’re taking privacy regulation very seriously.

There are private channels in the Cosmos server with sensitive information, what about those?

  • as the tool is based on a bot, the discord admin can simply not give the bot access to the sensitive channels and that data won’t be collected.

What sort of data does the tool collect?

  • messages data with a focus on interactions (replies, threads, emoji reactions) as this allows us to understand the shape of the network (the organizational network analysis). And we also collect perception data through the pulse survey (respondents can choose not to reply). We do not collect any financial data nor health records data, we won’t ask you for your name, wallet, date of birth, nationality, sexual orientation nor gender.)

let us know if you have any other questions :slight_smile:

1 Like

I am very much looking forward to a testcase of this setup. What kind of information and how it is shown in a live dashboard is something I want to play around with to get a feeling of how it can be used and what we can find with the tool.

I believe it is a field which might become much more important in the longer run when we progress, Cosmos and crypto become bigger. Having some kind of metric which shows community health or sentiment (gem of scam) will allow people to have a better judgement whether they want to join a certain project. Or community managers who see the numbers deteriorate have the tools to react earlier.

2 Likes

I tend to agree. I think it’s incredibly useful, especially as the organizational network becomes more complex.

will the entirety of the collected data be publicly available?

The passive data is already publicly available as it’s the very messages in discord. The survey data needs to be private, so we’ll only share the aggregate and not individual responses.
Currently, only our data scientists (doxxed and with ethical data handling training) have access to the raw survey data (anonymized).

This is more for the future, but we’re also exploring a partnership with Maastricht University so their data science students can use the anonymized data for projects and provide insights to the communities. If the partnership comes to fruition, we’ll then invite Cosmos to opt in.

1 Like

You’re most welcome to come hang out with us for the UX/UI meetings. I’m also open to do 1-1 with those interested.
We’re hosting regular sessions in the RnDAO server, so you can just join and say hi in intros and I’ll orient you :slight_smile: :

1 Like

so this is a proposal for funding to develop surveillance tooling for the cosmos across platforms?
How do you plan to motivate participation? I dont find pulse surveys appealing at all. it sounds like you are pitching the addition of pop-ups to the user experience.

Your proposal is entirely unclear what your proposal is going to provide the cosmos. it defines Community Health = health of its community and its vibez…what does that even mean?

You say you are currently in week 5 of roadmap, then below, claim you are in week 9.

I dont think I would vote for increased survailance of the cosmos and its participants, especially when i dont see any basis or metrics to value potential success of the community health metrics you have yet to disclose or conceive.

Would agree with the above. What are some of the likely benefits for the Cosmos community? “Vibez” is great and all, but the sum you’re asking for is not insignificant. What would the Cosmos be able to do with the data that your project might be able to provide?