Rethink Charity

Powering High-Impact Charitable Projects

Donate

2017 LEAN Impact Assessment: Evaluation & Strategic Conclusions

2018-03-01 General


This is the third post in the LEAN Impact Assessment series.

This is the third post in the LEAN Impact Assessment series.

In the previous Quantitative and Qualitative reports we aimed to describe our findings with minimal commentary in order to allow the data to ‘speak for itself’ and allow readers to draw their own impressions without being influenced by our own interpretations and conclusions. In this report we aim to offer more interpretation and analysis of the implications of our findings for LEAN strategy and for EA local groups.

The LEAN Impact Assessment has aimed to provide insight both into the status and value of EA local groups as a whole, and the efficacy of efforts (by LEAN and others) to support these groups. These topics occupy the first two sections of this report, respectively, and in the final section we outline LEAN’s strategic plans formed in the light of these findings.

Our findings represent only a first step in researching EA Local Groups and we plan to conduct further, more specific, research into local groups and LEAN’s services in the future.

Executive Summary

      • EA groups report producing significant impact e.g. counterfactual pledges and donations influenced
      • Most members report that groups are a large or very large factor in their engagement with EA
      • Organisers express a strong demand for personal support and feedback, written guides and research on group impact

Group Size and Activity

The LEAN Impact Assessment offers the most comprehensive empirical investigation into EA local groups to date. Prior to the LEAN Impact Assessment there was little, if any, systematic data about EA local groups as a whole, though minimal related data was gathered through the EA Survey (also run by Rethink Charity). As such, the first stage was to provide basic information about the number and size of EA groups and their activities.

Our sample (disseminated with help from CEA and EAF) included 98 discrete local groups and a larger number of organisers and members. This sample likely does not include all EA groups. Nor did respondents answer every question. However, it seems reasonable to assume that non-respondents were, on the less, disproportionately less active groups or groups who had less impact to report for particular metrics.

Reported group size ranged from 1 to 1350, with a median of 10. This means that the reported group size numbers are heavily dominated by a small number of very large groups. Of a total 4280 reported group members, across all groups, more than half the reported members come from the largest 3 groups, almost 70% from the largest 5 groups and almost 78% from the largest 10 groups. By contrast, more than 50% of groups contained 10 members or fewer and slightly more than 76% reported 20 members or fewer.

As noted previously, the extent to which a small number of very large groups account for almost all EA local group members is likely somewhat overstated due to differences in how group “members” were counted by different organisers. For example, the number of group members reported by the largest group in our sample corresponds to the number of group members they have in their Facebook group, but their reported number of people who have attended multiple of their events is much lower (>1300 and 160, respectively). While it is possible that smaller groups may also be ‘over-reporting’ their numbers in this way, it does not seem possible that the smaller groups could be inflating their numbers to the same extent (unless we suppose that the majority of groups have 1 member or less).

Accounting for this, the gap between the typical group and the very largest groups is less astronomical, though still substantial. Of course, the number of group members is not in itself important, but rather the impact that each group can have, which follows later in the report.

EA Survey Data

It is worth briefly comparing this data with the results of the EA Survey on local group membership, to see whether they are plausible, and to gain insight from a distinct source on the scale and scope of EA local groups. We would not expect these numbers to match up particularly closely, given the different questions asked and very different sampling techniques (the EA Survey was distributed widely and answered by EAs and non-EAs alike and had several thousand responses), the Local Groups Survey was distributed directly to group organisers (as well as posted in relevant facebook groups) and was specifically for local group organisers and members. Nevertheless it seems worthwhile to compare the results across these surveys to see whether EAs in a broader sample report being influenced by EA groups, rather than in a sample specifically targeting group organisers and members.

The EA Survey found 469 EA respondents indicating that “Yes” they were a member of a local group, compared to 953 answering “No” and 430 non-responses for that question. This means slightly more than 25% of EA respondents (including those who did not answer that question at all in order to give a more conservative estimate) reported being in an EA group.
We cannot assume this proportion applies across the whole EA population though, since it seems plausible that those EAs who answer the survey may be more likely to be in an EA group (assuming they are generally more active and connected). On the other hand, the absolute number of EAs in local groups is likely to be higher than the absolute count reported here, since the EA Survey doubtless did not include all EAs. Regrettably, no-one knows precisely how many EAs (by any definition) there actually are in the EA population (or whether or how far EA is growing). Nevertheless, almost 500 self-reported EA local group members from the EA Survey sample of ~1800 seems an appreciable number.

The EA Survey also provided data on whether individuals reported that they would attend an EA group if there were one near their home. In addition to the 469 EAs who said they do attend an EA local group, a further 271 reported that they would if there was one near their home and a further 310 who do not attend suggested that they were “unsure” whether they would attend or not. This is suggestive of fairly significant demand for EA local groups from individuals who presently cannot attend one due to lacking one nearby.

The EA Survey also offers data regarding the importance of EA local groups in getting EAs into and more involved in EA. 3.6% of the sample reported first hearing about EA from a local group, slightly exceeding Doing Good Better (3.4%) and Facebook (2.8%). Due to the wide variety of different answers indicated in the survey few ‘routes’ received very high percentages (the highest were Personal Contact and LessWrong on 15.5% and 15.3% respectively, followed by ‘Other blog post’ and SlateStarCodex on 9.4% and 7.4%). http://effective-altruism.com/ea/1h5/ea_survey_2017_series_how_do_people_get_into_ea/ (Table 2)

A survey sample drawn specifically from the EA Facebook group (using different methodology) returned higher numbers reporting that they first heard about EA from an EA local group. Here 7% reported hearing about EA first from a local group, beaten only by Friends (15%) Peter Singer/TLYCS (13%) and 80,000 Hours (13%). Including the 5% who reported first hearing about EA from a university group, the total % hearing about EA from groups was 12% (much closer to the top options) and comfortably beating the other options, including Facebook, SlateStarCodex, LessWrong and Doing Good Better. (ibid)

These findings are suggestive of EA local groups playing a significant role as the first place that many EAs encounter EA, albeit as one among a wide variety of different sources each making up a minority of the whole.

The EA Survey also offers data regarding how many EAs thought that EA groups were important in getting them into or more involved with EA. Here respondents could select multiple options as each being important. 261 respondents indicated that local groups were important for getting them more involved in EA, whereas the most commonly cited factors GiveWell and ‘Book or blog’ were cited 532 and 519 times. 261 EAs in the sample getting more involved in EA seems like a significant source of potential impact. It is significantly smaller than the number citing other factors, but this is presumably somewhat due to the fact that almost every EA has probably encountered GiveWell or an EA book or blog, whereas likely fewer than 50% of EAs have encountered a local group.

Overall, we take this findings from the EA Survey to offer reassurance that EA groups are reaching and influencing a substantial number of EAs, even in a broader sample not specifically targeting group organisers.

Age of Groups

Readers may be concerned by the fact that so many individual groups appear to be so small (for example, 9/85 contain fewer than 4 members). Likewise, looking at various metrics of impact, it is reliably the case that a number of groups report little impact. If many EA groups are largely quiescent or inefficacious, then this may be a matter for concern.

To try to understand this better, we looked at how long groups had been running. A large number of EA groups reported being very new.

This may be reassuring context when considering why some groups have not had much of an impact. They may not have had much impact yet, but many groups have not even seen a complete academic year or two seasonal pledge drives. This may be a neglected feature to consider in EA movement building discussions: a large number of EA groups (see the left hand side of the chart above) are still in their infancy and in the coming years may be coming into their own.

If we look at the number of groups of certain sizes at different ages, we see that while a majority (68%) of groups founded less than a year ago have fewer than 10 members, this proportion declines substantially for older groups (38% for groups 1-2 years old, 33% for groups 2-3 years old, and 14% for groups 3-4 years old).

Similarly, the median group size increases with group age (we have excluded the 5-7 year old groups from this graph because with only 2 members in these categories, the median is fairly unindicative):

A consequence of this is that even though most groups are younger groups (58% less than 2 years old, >80% less than 3 years old), most members are in older groups.

Another possibility we considered is that EA group size and activity might be quite variable across time, with groups, for example, being moderately active with a number of members, then falling away when a core organiser leaves or in between academic years, and then gaining more members once an academic year has started. If so, then while at any one time (such as the snapshot our survey provides), a number of groups may appear to be all but non-existent, the same groups may range across time through periods of activity and inactivity. Our qualitative data also included city-based group organisers reporting that they had high turnover of young professional members who would stay in a city only a short period of time before moving. This is partly suggested by our data on the perceived ‘precarity’ of groups, with a significant minority (30/89) of organisers thinking their group was unlikely or very unlikely to continue to function after the current organisers left. This is a trope which we heard from a number of organisers, where groups actually fell into abeyance after previously being very active. We presently lack data on how common this is, but this, and how to handle transitions better, as well as how to keep groups could be kept active, is a potential topic for future research.

Impact

Calculating the impact of local groups is exceedingly difficult in a number of ways. EA groups aim to have an impact in a variety of different ways: attracting new members to EA, encouraging members to become more engaged with EA (through providing social connection, motivation and information) and (directly and indirectly) encouraging members to take more effective actions (such as donating to effective charities, taking the pledge, considering their career based on EA principles), and increasing member retention.

Some of these things are intrinsically difficult to measure and quantify. For example, many EAs report that being in a local group increases their motivation and engagement with EA. Some impacts which we think EA groups are likely to have, are all but impossible to measure. It seems plausible that EA groups contribute to retention of EAs, as some EAs report that they likely would have left the movement but for their involvement in a local group: but EAs who do leave the movement and their reasons for leaving are typically inaccessible to EA data-gathering, and people who never join the movement but would have, counterfactually, had they encountered a local group are a fortiori inaccessible. As such, much of the true impact of local groups may be excluded from our analysis.

Further, it is exceedingly difficult to discern the counterfactuality of each of these outputs. While we can ask members and organisers for their self-reported estimations of what they would have done were they not in a group, it will often be hard for them to genuinely know. Many report that they would not have taken certain actions were they not in a local group, but counterfactually, it is possible that they would have been moved by something else to, for example, take the pledge had it been impossible for them to join a group. Conversely, an individual in a group may feel confident that they would have continued to take effective actions if they weren’t involved in a local group, but it is possible that they may be mistaken about this. This is particularly difficult given the wide variety of influences on any given EA’s actions: their interactions with EAs face to face in a local group, their contact with other EAs online, their reading EA literature, their attendance at an EA conference, may all influence their EA actions and attitudes and which of these factors are important may vary substantially across different individuals and the influence of all of these factors may not be transparent even to the individual in question.

Likewise, EAs may reasonably disagree quite radically about how much value to attach to each of these things (for example, the value of a given person taking the Giving What We Can pledge) or getting an additional person involved in the EA movement. We do not aim to settle such fundamental debates here, but rather report figures for a variety of metrics which will be considered of value to a wide range of EAs with different values.

Below we report figures for the collective ‘outputs’, for various measures, across all the groups in our sample:

Clearly these measures cannot capture all the impact of EA groups, since many potential effects of EA groups were not or could not be measured by the survey e.g. less tangible benefits resulting from reducing EA attrition, value drift or of connecting EA group organisers to valuable opportunities.

Even looking solely at the estimated (by group organisers) counterfactual impact of local EA groups, these figures seem quite substantial. As, at present, most EA groups are run by unpaid volunteers with little or no funding, the direct costs associated with local groups are quite low. The costs associated with volunteer or paid staff time, are likely to be larger, but here there will be higher variance. For many organisers the opportunity costs of their time spent running an EA group may be very low and there may even be net personal gains from their involvement in terms of experience, career capital, boosts to motivation and connection with others in EA. (http://effective-altruism.com/ea/1fh/lessons_from_a_fulltime_community_builder_part_1/cea) Of course, for other EAs, with higher impact elsewhere, running a group would be a net loss. We expect that, for the most part, individual organisers may be best placed to make these judgements themselves. We do, however, think that further research into the activities and opportunity costs of EA group organisers would be valuable.

Having looked at the total outputs across all EA local groups it is, of course, important to try to look at this more at an individual group level. The mean impact per group continues to appear to be quite high, but these numbers are driven upwards by a small number of very successful groups.

Therefore we present the median figures across a variety of metrics:

While these figures are much lower than the figures reported by the most successful groups on each metric or the mean figures, they still seem plausibly to represent significant impact. As noted, it is impossible to attach a definitive value to these various outputs, but 3 counterfactual GWWC pledges, and 5 people becoming counterfactually actively committed to EA may represent significant impact.

Of course, looking at the median figures across different metrics only captures one facet of the distribution of impact across different individual groups. If one wants to get a picture of how much impact a ‘typical’ active EA group’s impact, one might think that the median figures under-represent this, given that it includes a number of groups who are very new or quiescent (having one organiser, but no group). For example, if you want to know how much groups typically raise through group fundraisers, you might want to exclude from these figures the many groups which didn’t even attempt to run a fundraiser. By contrast, if you want to estimate the expected value of starting a group, this might be a more appropriate metric; or you might want to identify a more specific reference class by looking at groups in comparable situations (e.g. in cities which you judge to be similar) and draw inferences from that. We are wary about digging into these different ways of representing the data too much, because it introduces too much freedom for judicious selection flattering results, and we recommend that the reader looks at the specific results we report in previous sections.

Is it all the largest groups?

One of the trends that appeared most clearly in the results we posted in the quantitative data report, was how far the total figures (e.g. for members or money raised) appeared to be dominated by a small number of ‘super groups.’ Much of our data exhibited a very strong power law distribution.


We think this is likely to lead to the impression that since it seems almost all the impact from local groups, as a whole, is coming from a small number of ‘top’ groups, this is where attention and investment should be focused. This is worth considering further.

Firstly, it should be noted that strategically this doesn’t necessarily follow. If we can, at very low cost (for example, time valued at $500), cause a smaller group to bring about 3 counterfactual pledges (valued at, say, 3 x $10,000 each), this may represent a better investment than trying to help a large group producing 100 pledges produce yet more pledges, as even though their mean impact may great, producing extra marginal impact may be very costly. Even if we suppose that the number of members in a group is all that matters (for producing impact), it may be that it is most effective to focus investment in the smaller groups, in order to increase their size to that of the larger groups. This may hold true for other metrics as well. Suppose we are interested in recruiting top EA talent. It may be more likely that talented individuals will come from larger groups (since there are more members), but it may be that such individuals would likely be found anyway (due to already being surrounded by and connected with many EAs in an EA Hub). It might therefore be more important to support smaller local groups in connecting EAs with promising opportunities, as they are more likely to be missed. None of this is intended as an argument that smaller groups actually are more important to invest in, merely that it does not automatically follow that if certain groups produce most impact, it is more important to attend to and invest in these groups relative to the more numerous smaller groups.

However, there is also more to be said in terms of understanding the data and which groups are producing most impact. Given the striking distributions of group size and various measures of impact describes above, it would be tempting to conclude that it is the largest groups (with the most members) that are producing most of the impact. Further, one might suspect that it is largely simply having more members that drives increases success on various metrics (more counterfactual pledges, more donations influenced and so on).

Further consideration of the data suggests that this is only somewhat true. As noted in the quantitative report, for most measures there was a positive correlation between number of group members and various outputs, however this varied. For example, there was a strong correlation between numbers of group members basing their career on EA principles, but little relationship between the size of a group and the number of new attendees to events who were unfamiliar with EA.


Moreover, as we noted in the quantitative report, the data did not suggest that larger groups were better at producing outputs from their members (e.g. getting members to make career choices based on EA) – in fact, there appeared to be a weak negative relationship between the size of a group and the proportion of members who were basing career choices on EA or taking the pledge (plausibly explained by the fact that larger groups may contain relatively more new members and smaller groups may contain, as a proportion of their total size, relatively more core organisers).

It is also important to note that these correlations and various positive outputs do not necessarily suggest that having more group members causes higher impact in terms of pledges taken, funds raised etc. It may be that a third factor (group activity, propitious environment, good organisation) reliably causes groups to be larger and to have more pledges, funds raised etc.

To gain more insight into these questions we looked at the highest performing groups across different metrics (e.g. number of members, amount raised in group fundraisers, number of members becoming actively committed to EA, number of pledges, number of career changes based on EA principles, and number of new attendees at events who were unfamiliar with EA). Just because each metric seemed to be dominated by a small number of groups performing exceedingly well on that metric, it didn’t necessarily follow that it was the same groups dominating across each metric: it might be that the groups accounting for almost all the total group members were different from the groups accounting for almost all the pledges, for example.

Our analysis suggested that there was quite a lot of overlap between the groups dominating across different metrics. We don’t name specific groups because we wish to avoid publicly ranking groups in terms of success. Of the 5 groups with the highest member count, 4 recurred multiple times (3-4) over the other 4 categories, with the largest group topping the categories for active commitments, pledges and career changes. Notably, the other group in the top 5 for size did not recur across any of the other categories.

However there was scope for divergence from this pattern of dominance by the ‘top’ groups. Across the ‘top 5’ for the other 5 categories, each time at least 2 of the top 5 didn’t appear in the top 5 for any other category (e.g. at least two of the groups reporting the most pledges did not report the most members, most funds raised, most EA career choices or most new attendees). Moreover, one local group appeared in the top 5 for all but one of the remaining categories (funds raised, active commitments, pledges and new event attendees) despite not being one of the groups with the most members (indeed, they only have 35 members, placing them outside the top 10 for size).

Two of the categories examined were also striking outliers. The top ‘group fundraisers’ did not include the ‘top’ group across most categories and 3 of the 5 places were taken by groups which were not among the largest groups. As this graph shows, relatively little of the variance here was explained by group size:

The number of new event attendees, for a given group, who were not familiar with EA beforehand was even more striking. Here none of the largest groups were among the top groups (though the largest group, tied for 5th place) and only 1 of the 5 groups recurred across any other categories. This is quite noteworthy because this metric seems like the best measure we have (from this survey) for groups reaching out to non-EAs and succeeding in at least introducing them (via an event) to EA, yet it seemed to bear little connection to success on any other metrics. One other striking feature of this category is that all of the top groups (except for the largest group, tying in 5th place, and with only half as many new event attendees as the top group in this category) were from non-Anglo-American countries. While this is purely speculative, an explanation for this pattern might be that these groups are aggressively reaching out to people unfamiliar with EA in their areas, getting them to attend events, but largely not seeing success in transferring this into increased group membership. This issue probably bears further research, as it seems plausible that EA groups outside of the traditional geographical areas may face distinct challenges and require more tailored support (such as translation of materials).

These findings suggest that though a small number of groups tend to be very successful across different categories, there is still clear scope for other groups, which are not particularly large to produce great impact, competitive with the largest and highest performing groups.

It is also notable that many of the groups in the top 5 for various metrics were not from obviously propitious environments (e.g. elite universities or major cities). The fact that there is a fair amount of variance in success across different metrics, including success outside of the largest groups and outside of obvious EA ‘hubs’, suggests that there are influences on group impact beyond size and being in propitious locations. Further research seems needed to investigate how the most impactful groups attain their success and to see how far this can be propagated as best practice.

A final observation worth noting is that the much larger influence of the biggest group(s) may be a result of particular, contingent policies, rather than an inevitable feature of the way that EA groups’ impact will be distributed. For example, EA London reliably dominates most metrics by a substantial amount, but is unusual among groups in having had a full-time paid organiser, since 2016 (and now two dedicated staff). Looking at it from the present vantage point, it may appear inevitable that London would have grown to the size and influence that it has now. However, despite having been running since 2013, until quite recently, EA London was much smaller, averaging around 5 social event attendees every other month in 2014 compared to slightly more than 50 at the end of 2017. Of course, not all of this growth should be attributed to the presence of a funded organiser, and nor does it suggest that an organiser would have been equally successful in a different city, but it does somewhat count against the view that certain groups were simply inevitably going to be very large.

Further Evidence of Group Impact

Above we noted the median results for various metrics of group impact (e.g. pledges, donations etc.). However, as noted, groups also aim to have impact in a variety of ways which are harder to measure and quantify or which simply can’t be translated into a median value per group. We note some of these here:

We asked individual organisers and group members to indicate how much of a factor being involved with an EA group was for their involvement with EA. As noted in the quantitative report, a majority reported that being involved in an EA group had been a “large” or “very large” factor for their engagement with EA. Similarly, 89% of organisers and 78% of members reported that the way they thought about the world and/or their behaviour had changed since becoming a member of a local group, with large majorities of these reporting that they expected to have more social impact as a result of these changes.


Though hard to attach a precise value to, and reliant on self-reports, this is strongly suggestive that local groups are having a positive impact on EAs and increasing their engagement with EA. While this is not direct evidence of impact, it seems likely that increasing people’s engagement with EA may lead to impact by, on the whole, making individuals more motivated and promoting EA actions as norms.

Our qualitative data also strongly supported this, with many individuals explicitly reporting the importance of “personal interaction” with other EA for their motivation and engagement. It seems plausible that, for at least a subset of EAs, face-to-face interaction with other EAs, rather than only online contact, may be important.

There was also a general indicator that local groups are valuable to EAs, as a large majority (93%) of members rated their group’s activities as “valuable” or “very valuable.” Interestingly, group organisers, while still having a majority (57%) rating their group’s activities as “valuable” or “very valuable” and 89% rating them between “moderately” or “very valuable” were noticeably less positive than group members. Over half (55%) of all members who responded rated the group’s activities “very valuable.” We speculate that this may be partly explained by organisers and members interpreting the question of how valuable their group’s activities are slightly differently, with members taking it to be about how valuable the activities are to them and organisers taking it to be asking for an evaluation of how valuable the group is as a whole. It is also possible that organisers felt the need to be more self-critical about their group’s activities and achievements, even though members near-uniformly found the groups to be extremely valuable.

Groups also introduced a fairly large number of individuals who were unfamiliar with EA to EA via events (nearly 6000 in total, and a median of 30 per group- which is quite substantial given a median group size of only 10). This cannot be directly identified with impact (as we do not know what proportion if any of these individuals became more engaged with EA- this would be an avenue for potential future research). However, there is suggestive evidence from our findings that slightly more than half (138/254) of our respondents did not consider themselves an EA prior to joining an EA group. This may be suggestive of non-EAs joining groups and coming to identify with EA, though it may also reflect individuals applying a very stringent definition of EA to themselves and only identifying as EAs once they have become actively involved.

Value of existing programme services and resources

Where identifying the impact of local groups is, as we note above, exceedingly messy, identifying the impact of efforts to support local groups is substantially messier.

Where local groups aim to produce impact through a wide variety of means: introducing people to EA, making EAs more motivated and engaged, making them more informed and connected to opportunities, encouraging impactful actions, increasing retention and reducing dropout, efforts to support and promote the work of EA groups in a similarly broad set of ways with a broad set of means.

For example, LEAN has aimed to support EA groups in a host of ways:

      • Providing technical infrastructure services (e.g. group e-mails, group websites, meetup.com etc)
      • Providing informational resources (e.g. guides to running groups, the map of EA groups)
      • Personal support and feedback to group leaders
      • Supporting group communication (newsletter, group calls etc.)

Taking even the more straightforward of these services – providing a group with e-mails and websites – these services may potentially have positive effects on a host of outcomes via a variety of different mechanisms. They aim to make the group appear more professional, which may attract more members and/or make the group (or wider EA movement) seem more appealing, and it may allow the group to access opportunities (in virtue of as one of our qualitative respondents put it “showing people that we’re not some fringe thing that’s only locally run”). They may also make organisers and current members feel better about their group, increasing motivation. Providing them as a service may make running the group more convenient and less costly for organisers (who might otherwise feel they need to set up these solutions themselves), encouraging people to run groups and increasing retention. (For diverse other positive effects of support offered to local groups, see the qualitative report).

However, for each of these putative mechanisms and metrics, there are innumerable other factors influencing how professional an EA group appears, making EA groups more or less convenient or costly to run, and influencing how many people are attracted to or engage with the group. And likewise for every other means of supporting groups: providing leaders with mentoring or guidance, making written guides available, providing video calls for leaders to discuss issues and so on. Counterfactuals here are, of course, particularly difficult to discern, given the mutual influence of so many different factors on the same outcomes.

As such, identifying the causal impact of any particular service on the performance of groups (whose own impact is itself very difficult to identify, as we noted in the previous section) is all but impossible. As a consequence, in this report, we rely on the reports of group leaders on how useful they found different services offered to support groups. These evaluations certainly have their limitations, reliant as they are on the self-reports of organisers, but we think they are among the best evidence available to us of whether services are actually helping support groups.

Alternative Analytic Strategies

It is worth briefly sketching out some alternative means of calculating the impact of efforts to support local groups and why they are either unworkable or severely limited:

      • Randomised assignment of local groups to receive support or not (for example, one set of groups might receive websites or personal support, and another set would not): this is a gold standard of experimental design highly familiar to EAs. It may be impractical, in this case, to aid some groups and not others though. Given very small samples of highly heterogeneous groups, it may be very difficult to ensure that the treatment and control groups are comparable. Furthermore, some services provided are easily accessible public goods (e.g. online resources and guides). Also not treating a ‘control’ group may have its own effects: e.g. severely dispiriting group organisers who are denied assistance or encouraging organisers to adopt their own ersatz solutions (for example, if the treatment group are, very visibly, being given new websites, this may move those assigned to the control group to set up their own website).
      • Comparing pre/post treatment metrics: for example, seeing how a group performs before/after receiving support. We already have some data along these lines: for example, groups reporting increases in response rates after using an official-looking group e-mail address. We aim to expand this kind of small scale experimentation, where practicable, next year. However, in many case, where there are not discrete metrics (such as e-mail response rates), this will also be limited, as multiple other changes over the course of a year will confound the changes observed after the implementation of a particular service.
      • Comparison of outcomes from observational data for different kinds of groups e.g. CEA groups vs LEAN groups: this particular suggestion would be essentially impossible, because many groups receive support from a number of different sources, meaning there are few, if any, “pure CEA” or “pure LEAN” groups. It may be possible to make informative comparisons of groups of other types.
      • Comparing EA individuals who are or are not in EA local groups (based on the EA Survey) to see their impact, rates of attrition etc. This suggestion seems hamstrung by the fact that these differences would likely be severely confounded. Individuals who are (or are not) in an EA group are likely to systematically differ in other ways due to self-selection, among other things. For example, more motivated individuals may be more likely to attend an EA group. Alternatively people who are not in an EA group may be more likely to live in isolated areas away from an EA hub compared to those in a group, which may independently exert an effect. Efforts could be made in an analysis to ‘match’ comparable people and overcome this in other ways, however we are not sure this would be reliable or worthwhile. (The EA Survey data is freely available to anyone who wishes to make an attempt.)

Evaluations of Group Support and Services

During the interviews and the Local Group Survey, we received feedback about a range of resources and services, some of which had been provided by LEAN exclusively and some of which had been jointly offered by LEAN and other EA organisations or independent EA individuals. Disaggregating the impact of different organisations’ services is therefore difficult to do systematically, though it is sometimes possible to identify individual cases of impact from the qualitative interviews. Indeed, often group organisers themselves couldn’t identify which services were provided by which organisations. Nevertheless, we think these results offer a reliable indication of the value which organisers attach to different services and thus, at least to some extent, of the degree to which they are likely to help support EA groups’ work.

Technical support

Technical support is the area where we can connect respondent experiences most directly to the LEAN’s investment. This is because, during the time frame considered, most of these facilities were not made available by any other organisation. Some organisers supplied their own websites and Meetup.com accounts, but these are in a minority. Furthermore, organisers came to LEAN over time and arranged for us to take on hosting for existing websites, or payment for existing Meetup.com accounts.

Our quantitative report offers ratings of the usefulness and impact of various of the tech support services. Combined with our qualitative data, these generally follow a pattern of services being found to be highly useful by a small number of groups, but not very useful or not used at all by others. That said, “technical support” as a whole “for instance, subscriptions for online services, free websites, group email addresses” was rated as useful or very useful by >70% of respondents. Qualitative responses bore out a pattern which recurs across our tech services, which is that these services were invaluable to some groups and minimally or not at all valuable to others, with a large number of groups not using the service at all.

Websites: organisers were asked whether their group website “makes a non-trivial difference in the effectiveness of your group’s outreach efforts” and websites were rated as “significantly useful” rather than “no more than trivially useful” by a slim majority of respondents. Response rates were low though (39/98), in line with the fact that relatively few groups have a website.

Meetup.com subscription: We asked organisers to estimate how much of a counterfactual increase (%) in attendees they had as a result of using meetup.com. There was an exceedingly wide range, with estimates literally ranging from 0% to 100%, with a median of 15%, but a small number of groups gaining very large increases (50-100%) from using the services.

E-mails: we did not directly survey individuals about the usefulness of group e-mail addresses, but some indication of interest came from the fact that a majority (29/54) respondents indicated that they would like one for their groups. Qualitative data offered further confirmation, with a number of organisers noting that they preferred having a group e-mail as it looked more “professional” and that they noticed higher response rates after using them.

We are not surprised by the fact that these services were extremely valuable to some groups and of little value to others, since we find that EA groups differ extremely in a wide variety of ways. As such, a tailored approach may be necessary to direct specific service to the specific groups which find them useful. It does not automatically follow from the fact that many groups find relatively little benefit from using meetup.com, that it is not worth continuing to provide it if a small number of groups gain substantial benefits from it. This is especially true in cases such as Meetup.com where it is cheap and easy to provide accounts to groups who want one, and where the service allows three different groups to share one subscription.

Personal Support and Expertise

LEAN provides expertise and support to groups through the following services and activities:

      • Personal Feedback
      • Practical support and ideas
      • Written guides
      • Connecting and introducing
      • Impact assessment and research support

The LEAN Impact Assessment quantitative report found that group organisers attached extremely high value to receiving personal feedback, practical support and ideas and guidance in the form of written guides.

Personal feedback and support: 78% useful or very useful
Practical support and new ideas for group activities: 94% useful or very useful
Written guides: 83% useful or very useful

Our qualitative data filled out the details about exactly what and why organisers found valuable. A recurring theme was that some organisers felt insecure about running a group, as to whether or not what they were doing was effective, felt isolated from the broader community and felt that personal contact for feedback, reassurance and advice was beneficial for motivation.

Increased guidance and provision of EA materials (for example, stock content which could be posted to their Facebook group) were also cited as things which would make running a local group easier. Some organisers reported that they felt that they were having to invent things themselves, even though they knew that other groups must have worked on similar problems before.

Other related factors which came up in our qualitative data, but which were not captured in terms of our quantitative metrics, involved helping connect EAs and groups with relevant people (either directly, through our networks, or indirectly through the use of the Effective Altruism Map.)

A further recurring theme was a demand from group organisers for more research which would inform how they should be optimally running their groups. Many EAs wished to know whether they were acting effectively and having an impact, and wished to see centralised data from other groups to better discern what would be effective.

Group Communication

      • Group Calls
      • EA Group Newsletter
      • Mentoring Programme

LEAN, in some of these cases in collaboration with CEA and EAF, support a number of platforms for EA group organisers to discuss with each other and with orgs supporting movement building, to share expertise and information. There was substantial variability in how useful each of these platforms was rated as being, as some were new or little used or organisers were not yet aware they existed, whereas others were widely valued.

Group Calls

Slightly more than half (54.3%) of respondents rated group calls as being useful or very useful.
Qualitative data suggested that organisers were broadly supportive of measures to improve interconnectivity between groups and group leaders, though there was little specific mention of group calls. Furthermore, some interviewees mentioned that the one-size-fits all nature of group calls meant that they had not found the issues discussed particularly relevant for their own groups.

EA Group Newsletter

LEAN published eleven newsletters for group organisers between 2015 and May 2016. The group organisers newsletter was relaunched as an inter-organizational service during 2017, and four editions have been published (note that this is not the same as the general EA Newsletter, which is also provided in collaboration between CEA, LEAN and EAF).

Feedback on the concept is limited due to the fact that few respondents were in receipt of the newsletter. 32% indicated that they receive the newsletter, whereas 61.6% of respondents expressed an interest in being added to the newsletter. 53% of respondents to a question about the usefulness of the newsletter rated it as useful or very useful, while 40% rated it as neither useful nor useless.

EA Mentoring Programme

A pilot mentoring programme was initiated by CEA in late 2015. In 2017 the concept was rebooted by LEAN in beta form. As with the rebooted EA Groups Newsletter, few participants in the sample would have direct experience of the Mentoring programme so far. At of the time of writing, 9 mentoring pairs are included in the programme. Given this, it is not surprising that the majority (61%) of respondents rated the programme as ‘neither useful nor useless,’ though 30.7% deemed the programme to be either ‘useful’ or ‘very useful’. We received two or three comments from survey participants explicitly stating that they had not heard of the Mentoring Programme and the Newsletter, and asking to be sent further information. We will be following up on the pairs of individuals involved in the mentoring program to investigate how impactful it is.

Strategic Summary

This section summarises the strategic plans that LEAN has for 2018 based on these findings.

Tech support and group communication services will continue, via more streamlined techniques. Our tech resources will be focused on developing a high quality interface for written content and resources. In response to demand from organisers for these services, we will expand our investment in offering personal support and feedback to organisers, and will consolidate our online resources for group leaders. We think that these written guides and personalised support will be synergistic with LEAN’s expanded work researching EA groups and how they can best produce impact. More broadly, we hope to work more on discerning the specific needs of groups in different contexts and offering tailored support and advice.

Tech support

Having found that our tech services are highly useful to a minority of groups we have decided to continue, but streamline our service provision. While it might seem that services which are useful only to a minority of groups are not worth providing, we have concluded that the high level of usefulness to a small number of groups justifies providing these services. A significant consideration is that, in many cases, if LEAN ceased providing these services, the financial and time costs would likely simply fall to individual organisers, making running groups more costly for individuals. We have, however, switched to employing simpler, more automated processes for providing services (for example, a static site generator for EA group websites, and an improved email host) which requires less time and financial investment. At the same time we have also invested more in dedicated tech staff to ensure our systems are more reliable in the future.

Tech development

Based on the evident popularity of existing written content, and the widespread wish to see existing content streamlined, we are investing in a new web interface for organisers. This will involve editing existing community generated content, and assimilating it into a central, visually appealing interface. The new interface will be based on the EA Hub, which will itself be modernised and restructured. Where possible, we will facilitate editorial additions from the community, in order to make the tool a logical home for sharing resources.

Group Communication

We will continue to facilitate group calls alongside other groups, as there is still demand for these, though we will work on promoting more targeted calls (and, in particular, offering groups more individual calls where requested). We will continue to run the group newsletter (which now has a larger number of organisers added) to keep organisers up to date on relevant development and to help facilitate the mentoring program for interested organisers.

Personal support and feedback

In response to very high ratings of usefulness of these kinds of services in our survey, we have decided to dedicate more attention to providing personal support and feedback to organisers where requested. Much of this will take the form of one-to-one calls with individual organisers to talk through challenges their group is facing, connect them with relevant individuals or resources. We hope this kind of tailored support will help guide organisers, and also increase their motivation and confidence in their actions. This will also allow a closer determination of i) how much counterfactual impact groups are having, ii) how far particular services are supporting groups.

Written guides and resources

Similarly, in response to a strong indication of organiser interest in this, we will dedicate time to consolidating and adding to existing written resources aimed at helping organisers run groups. This will contribute to the new content tool for organisers on the EA Hub, mentioned above. We aim to update these resources on an ongoing basis based on our ongoing research into how EA groups are functioning and how they might function better.

Research

Many organisers expressed an interest in using metrics to determine their impact and in having access to evidence-based research on how best to run EA groups. We have also noted, throughout this report, various questions about EA movement building where further empirical research is needed. Some of this research may be conducted in concert with the EA Survey (also run by Rethink Charity). We aim to help individual groups to measure their impact, and to produce general research regarding EA groups for the wider community.

Grants

Recognising that funds are important for groups’ success, and having heard from a number of group organisers that lack of funds was a bottleneck, we are planning to make small grants to groups on a case by case basis to help facilitate group growth and are exploring ways to collaborate with CEA to direct funds to groups. Where appropriate we will align this with our ongoing research into EA groups, to try to discern how far targeted grants boost EA groups’ impact.

Acknowledgments

This report was analysed and authored by Richenda Herzig and David Moss.
Special thanks to Tee Barnett editorial feedback, and for coordinating those involved in the Impact Assessment, both internal and external. Further thanks to Peter Hurford for input as an internal advisor.

We are very grateful to Greg Lewis for his ongoing input as our external advisor.

Share post:

2017 LEAN Impact Assessment: Qualitative Findings

2018-01-05 General

This document is the second in the LEAN Impact Assessment Series, briefly summarising relevant data from our qualitative interviews with 31 EA group organisers. Click here to read the previous report from the series [1]. The third and penultimate report will synthesise data from qualitative and quantitative reports in order to derive general conclusions.

Table of Contents

Organiser Experiences and Wider Issues

Specific Services and Resources

Websites
Meetup.com
The EA Hub

Conclusion
Acknowledgements

The report is divided into two sections. The first section presents crucial themes from the interviews relating to common experiences, needs, or concerns experienced by group organisers.The second section presents data on existing forms of support offered by LEAN and other EA organisations supporting groups. In both sections, general observations are made followed by demonstrative quotes from the interview transcripts [2].

Organiser Experiences and Wider Issues

Appetite for Assessment and Evidence Based Guidance

There is a high level of demand among organisers for formal research into outreach and group management strategy. Organisers also want to be able to effectively monitor their own progress and impact, and they look for certainty and empirical backing to inform aspects of group leadership.

“I would really appreciate help with planning ahead and deciding on useful metrics to measure, so that we can assess how much impact we’re having.”

“Sort of like a roadmap too… like based on other groups and what they’ve experienced. Like first they have 1-2 years doing x, y, z and then they reach n members, etc…”

“Measuring impact for groups is so difficult, it takes so much time to plan. It would be helpful to have a vehicle through which to do this. So LEAN tracking metrics across groups would be very valuable.”

“What would be good is some plans. Like how should you organise a small group, a large group etc. How big are other groups? What is the average size?”

“The things you are doing, like the survey, it’s really something super important. We were thinking of iterating that here on a local scale, but then the person in charge of that concluded that this survey you did was good enough… it’s just something that we needed, and this is giving us more insight that we can chew upon.”

“I would be really interested in what you’re looking at, how you ask people… how you set up evaluation and surveys after events you’re holding, and stuff like that.”

“The most important is research on how to advertise…for example Facebook advertisements. What kind of keywords to write, and is it worth it? How many people do you get that way? How well it responds to persistence and all these things…”

Legitimacy and Reputation Concerns

In some contexts, organisers feel that there is a functional need for public legitimacy for their groups.

“It would be the idea of us drafting guidelines and having the government department of charitable giving sort of rubber stamping them. With something like that I feel… if there’s access to more prominent people in this field, we could send them Singer’s talk. But I don’t know if there’s any additional way of getting legitimacy.”

“We had more funds by using The Life You Can Save funds for giving games, and also more credibility.”

“I know I needed a lot of tech support setting up the eahub.org email address, but people respond to that better than something that has gmail at the end of it. We’ve noticed we have higher response rates.”

“[LEAN] made us a club website which looks very professional. And that went a long way to showing people that we’re not some fringe thing that’s only locally run.”

The Significance of Context

A crucial insight from the interviews is the degree to which group-specific context shapes and determines the success and opportunities available to organisers. There is variety in the demographic that organisers are able to attract, the culture and attitudes prevalent in specific countries and regions, the proximity of groups to other EA organisations and communities, or to otherwise like-minded collectives, to name but a few distinctions.

For this category, summarised examples are a more concise means to demonstrate this than verbatim quotes.

  • A group that raised a significantly high sum through group fundraisers benefitted from a proactive student body, generous state support, and an unusually high national culture of altruism and charitable giving.
  • A group in a large, spread out city found regular attendance hard to secure due to the travel times required.
  • One group has a strategy of targeting corporations in order to take advantage of a state requirement for corporate charitable giving.
  • The Pledge is less appropriate for groups in poor countries.
  • Several groups are benefitted by residing in cities with unusual concentrations of academics or experts. For example, policy experts in Washington D.C., Geneva, Brussels or London. Researchers in Oxford, Cambridge and Ivy League institutions and so on.
  • A group in the Middle East has to operate in the face of state limitations on charitable giving, and is looking for ways to partner with mosques and existing religious traditions surrounding philanthropy.
  • Several Asian groups are partnering on translation and reframing core EA ideas to target local attitudes.
  • Groups in some countries, like Germany, have to be more sensitive about association with utilitarianism or Peter Singer due to public hostility.
  • 80,000 Hours recommended careers are significantly less accessible in some countries than in others.

Group Influence

While tangibly gauging impact is a widespread challenge for organisers, some respondents mentioned concrete instances where a person either became actively involved in EA as a result of being introduced to EA in a group, or cases where getting involved in group management led to individuals engaging more strongly with EA.

“I got into EA through my local group at [University], which explains my interest in it. It’s changed a lot of my values, and I wanted to create that opportunity for other people.”

“It’s definitely helped me grow more, just taking those actions. [Member name] taught me as well, as it was because I got him into EA that he then became very involved.”

“Because I am organising, I do think the possibility is high that [nearby regional EA Group] is where I can do a lot. And so I certainly consider that in my career path right now… I plan on going to [regional EA Group] and working there in the future. So you can see, EA has changed my life a lot.”

“I think that they [group members who took on leadership after the respondent moved on from organising the group] became more engaged, but I don’t know if they would have done that some other way instead. Like maybe going up to [EA Capital City], or more conversations online, or skypes with people. Certainly a couple of people like [Name] and [Name] are much more involved than they were before and they took the Pledge while we were there. And we had a lot of conversations about these things…”

“The most useful feedback I received was locally, from the people who actually came to events. Seeing them be happy also made me happy because we’d spent time talking about all sorts of brilliant stuff that we couldn’t talk about with anybody else. Or actually convincing the first person to actually donate money, or work for an effective cause, or convincing someone in an argument that other people were thinking about or actually acting upon… this sort of feedback was still the best thing I received. And I think the best thing about my job right now is that I’m seeing this. So every time I organise an event I’m getting more and more people into this, and this is making me feel amazing.”

Organiser Insecurities

Some of our respondents experienced insecurity in their role as group organisers, especially those who had recently started up. For example, we encountered abstention from interview participation arising from discomfort in answering questions about group progress. A few participants also expressed concern prior to interview over whether or not their groups were large, active or successful enough to ‘count’. In conversation, respondents occasionally unfavourably contrasted their situation to an ideal, with the counterfactual typically being a larger, more active or more professional group.

“I feel a lot better about undertaking this with a team, but I don’t think…I don’t know that any of us have done like formal outreach or community growth in this sort of way before.”

“It’s basically me and a very small team, and we are still learning and none of us are really researchers in one of the EA organisations.”

Not only were insecurities indirectly apparent in how organisers spoke about their groups at times, but there were also some direct reports of specific anxieties and doubts, often related to respondents’ concerns over their own personal EA credentials, or the fidelity of their approaches.

“I hope that [University] group gets going, but I also just feel like an old guy trying to make stuff work.”

“Some EAs are vocally saying we shouldn’t be doing much outreach… we don’t want to dumb down the message. I worry… I think we are doing the right thing, but there are mixed messages out there about what we should be doing. I don’t know what people think about us and what we should be doing. I feel lacking in confidence. I kind of want a seal of approval that makes me feel like I’m doing the right thing.”

“There’s just so much literature, Jesus! I started reading about, you know, Peter Singer, and now I’m into professional finance… and I just can’t keep up! I had it in my mind that maybe I could get a position in some EA organisation. It’s a bit intimidating, though, seeing the positions and the people working in them, like ‘what the fuck? How did they do that? They raised $2 million?” Some of these people seem so alien…so far out from normal people.”

Personal Connection and Motivation

Personal interaction with other EAs is highly significant and sometimes critical for the motivation of group organisers.

“If I’d been without personal interaction, I’d never have gotten that much involved with EA.”

“I always seem to be inspired after I touch base. Because it can be kind of challenging when you’re not really immersed in a culture where people have these kind of ideas. So I always find it very uplifting to chat to people who are involved in the overall EA community.”

“I don’t even know why, but meeting all those people in there [EAG] and getting all this awesome advice that I really needed and feeling the sense of the international community rather than a local thing that’s generally myself and a couple of guys I’ll see each Thursday. I think this really helped me go and invest more, basically.”

“When it comes to feedback from the wider community, I still have this fuel from the 2015 conference. I’m still fueled by that experience. And I’m going back to London – I just bought a ticket today – I decided I’ll just go there and do it again. So in terms of that feedback, there would be no initiative from my side if not for that feedback.”

“After last year I felt a bit not connected. I felt alone here… yeh I told you how this guy just disappeared. But last year when I had the retreat, I met these wonderful EAs…”

While some of the organisers we spoke to were independent ‘self-starters’, most success stories referred back to critical connections, such as meeting people at a conference or retreat, or finding an equally committed co-organizer. Conversely, isolation from the wider EA community and a lack of equally motivated partners was one of the most frequently named challenges facing organisers.

Desire for Integration with the Global Community

In addition to the personal needs of organisers, respondents widely expressed a desire for better avenues to connect their members to the wider community.

“I’m kind of the only EA that has met the others outside of [country]. I kind of want members to have the chance to skype with other EAs.”

“I really want to see the club get more directly involved and engaged with the local [city] EA community. So, if you’re like a club member at EA [university] you pretty much automatically know most of the EAs in [city]. I kind of want that to be an automatic, organic thing.”

“I’d just like us to be more connected to the greater EA community.”

“Things that really help to implement would be to make sure that we can send more people to EA Global…that would be something we’d give out like candy if possible.”

In particular, some organisers found themselves at a loss for where to direct their members in order to integrate them more fully into the movement. Several expressed dissatisfaction with existing channels such as EA Facebook groups and the EA Forum with respect to bridging the gap between seasoned or professional EAs and newcomers.

“The EA Forum doesn’t seem like something that can be used at all by people who aren’t running an EA org or researching full time or whatever. It doesn’t seem like a terribly nice platform to recommend people to go look at.”

Direct Action

A related but distinct concern from respondents was the need for direct action opportunities in order to improve member retention. In some cases, EA groups appear to thrive on discussion alone. But in many cases, people need something more tangible.

“Around the world there is a common problem, that after people get involved in EA there’s not many things to do or keep them interested…generally I want to let people know that joining EA isn’t just about talking, but actually getting involved in things and improving yourself gradually.”

“People will say ‘well great, this sounds like a good idea, but what is there to actually do?’ It’s just a small community of people that are interested, and I think what’s tricky is that people don’t really see what they get from joining. And it’s like ‘ok, great. I like this idea. I’ll read all the materials, but why do I need to come?’ Kind of thing…”

“We’re getting better at spreading EA as information and making people interested in it, but we get many messages from people wanting to be involved in the club and actually do stuff. But then, you know, the stuff we do is more host meetups, talk about stuff, get speakers, or we talk about donations.”

“We got a lot of students and young professionals who liked the ideas, but because we don’t have anything concrete for them to engage in this…I can’t just say ‘Hey! Job opportunity at ACE! Apply for this!”

“We still look for ways of generating appeal. I think the prioritization project and influence over a certain number of dollars was useful for that. But since we’re largely focused around meeting weekly and talking about philosophical issues it’s not always as prestigious as joining another society that potentially has more to offer.”

“There are people that want to talk about weird stuff and ethical paradoxes, and then there are people saying ‘lets do something tangible as a group’. And so I’ve been trying to figure out how to make everyone happy…It would be cool to be involved in something that ties us to the community and helps people stay invested.”

Overall, discussion and social interaction are often insufficient to persuade members to regularly engage and commit to an EA group. Organisers on the lookout for group actions that offer tangible impact, as opposed to traditional mainstay group targets like career changes or pledges. Respondents also felt that activities need to offer goals which new EAs could not reach equally well by going it alone, if groups are to retain these individuals.

Productivity and Accountability

In many cases where organisers succeed in attracting a core group of regular attendees, they still struggle to find effective strategies for motivating their members to adhere to commitments and to take timely action on group goals. This is compounded by the fact that groups are organised voluntarily, which makes organisers reluctant to put pressure on their teams.

“Getting organisers who have enough time. I do organising, but I’m really busy. I want others to step up, but I don’t want to pressurise them. Others don’t care as much as me. Sometimes people offer but then bail, or don’t deliver.”

“There’s a challenge of commitment. When we have administrative talks we’ll say ‘right let’s do x, y and z’ but then one week, two weeks later, nothing has been done… And something else that’s important is to have accountability partners, accountability checks. We have this kind of…everybody’s a volunteer, so nobody takes a stand, you know? We talk and talk but the action gets blurry. Who does what and when?”

“It’s hard to say ‘alright, here’s the thing we need you to do. We won’t pay you for it, we just want you to do it in a specific way…can you do that consistently for a semester and then train someone else to do it?’”

“If me and, say, one other person were to go, I don’t think there would be a lot of willpower to really push to get things done.”

“I think it’s not that hard to find people who are interested in Effective Altruism a little bit, but those who want to be really involved, those are hard to find.”

“Entropy, as I would call it, was one of the biggest challenges. There was really a significant cost to getting people together and making them into a group. You know, an actual group… an organised group, a sort of quasi-organised group. So whenever I put these people together to work on a task they would quickly dissemble, they would lose focus, lose direction. I would lose direction myself, I would have to change my goals. So managing myself and managing the group, you know, without financial incentive, without the traditional ways of making people work together – as in a company or an actual organisation – that would be difficult.”

Not only do organisers struggle in finding committed members and soliciting productive output, but many also found it challenging to find time and motivation for their own goals and ambitions.

“I only hit a homerun once, and it was when the coach yelled at me and I felt it was unjust. So I had a lot of passion. And if you’d be willing to periodically say critical things to me, like ‘[name]! You’re not doing good enough!’ that would be really helpful.”

“Having consistent events at the same time and place would be helpful. I mean this is largely because I’m a bit… I’m not the most conscientious person, so I’ve just been doing events every few weeks.”

Across the board, organisers mentioned the challenges of balancing group administration alongside busy course loads and careers.

Specific Services and Resources

Written Guides and Resources

Respondents widely viewed written guides and resources to be highly valuable. Resources of this kind were often marked out as especially useful or valuable in helping organisers get their groups started, and in reducing labour; particularly reading and discussion plans and presentations. Respondents were enthusiastic about any resources which provided additional ideas for group activities, and for content creation services such as scripts to use for introducing EA to outsiders, and for content to share on social media. Unsurprisingly, the general EA Newsletter is valued by group organisers for this reason.

“I’ve gotten a lot of help via content from organisations. Any content on EA and GiveWell, and you guys [LEAN] also. The EA Forum is a great source of information. That has given me a lot of inspiration for meetups and stuff… It was very helpful.”

“Nowadays you can get everything, but two years ago a lot of stuff was not there. Now everything is much more organised, I think. Two years ago we had to do everything ourselves. There weren’t materials in German. I didn’t know all the presentations that were there.”

“I tried to put as much effort as possible into the meetings, but when I have resources to make that easier, that’s really, really helpful. So I think the meetings would have been a lot less polished and more casual without those resources, which is why I’m putting a lot of stock into those resources. I think I was able to get the core group really excited because of having like the reading list, for example. Just having that and being able to print that out…”

“One of the really important resources was the list of articles and things that Harri sent me that has been used for an EA curriculum in the past. We’ve modified that a little bit, but we’ve used that essentially to guide our discussions.”

However, this was also the area most frequently mentioned in reference to ways that group support could be improved. In particular, respondents felt that sufficient quality resources were available, but that organisation, coordination and presentation undermined the accessibility and use value of these resources substantially. Some respondents also expressed concern regarding the confusion caused by the existence of multiple organisations (e.g. TLYCS, LEAN, CEA, EAF, 80,000 Hours) involved in providing content and support.

“It’s really bitty and confusing, which people can do what… One obvious starting point would be: if someone wants to start a group… I don’t know how easy that is, but there’s a bunch of things to read and people to talk to, and it’s really confusing who’s going to be doing what. Clarifying this for new organisers… a tried and tested simple pathway…”

“Too much duplication… too much content in different places.”

“It’s clear from talking to [EA organisers in the country] that they would like a central point of of contact that coordinates with all the resources that exist in general.”

“I really get confused with all the things that have popped up connected to local groups.”

“I [organiser of a well known University EA Group] get a lot of emails from groups that want to start up but are lost and don’t know what to do. And I know there are some resources online for how to start your group and what to do, but there seem to be issues with finding that information, and navigating it, and making it useful. So I think informational resources could be made more accessible…”

“At the moment, what’s confusing to people starting a group is that they don’t know whether to correspond with CEA, EAF or LEAN. And I think that gives the impression that the community is not organised. Or that it’s a lot of fractured groups rather than a coherent… sometimes people post saying “I want to start a group, what can I do?” and they get about ten different comments from different places, and I think it would be good to focus on just one.”

In mid-2017, LEAN introduced a comprehensive spreadsheet, listing all known online guides and resources with support from CEA and EAF. Feedback on this resource map has been enthusiastically supportive, which reinforces the finding that improved presentation and organisation is sought after in this area.

“I think the most useful thing has been the resource map. If I could think of something that would be useful… it’s just there, which is really nice! I posted it on the conference event recently because I was like “This is amazing! Everyone needs to see it!”.”

“Whoah! This is great!”

It should be noted that, while the consensus stresses that better use of existing resources is a priority, respondents were not against the provision of fresh material in the future.

In addition, respondents had several recommendations for new resources or improvements to existing ones. There was a lot of interest in tools that would enable them to gauge impact. Additional content for use on social media was a recurring wish, as was the need for better support in finding speakers and locations. Finally, we received a recommendation to create a map that enables organisers to explain to the general public how EA organisations relate to one another and to local groups, and a resource that gives organisers concrete examples of how different kinds of support had been put to use by other groups.

“I’m also not sure if there is basic text for a group? Like basic text you can copy into a Facebook event and just slightly change. So basic drafts…”

“To have, like, things to post on the general Facebook page because inspiration runs out after a while of course. And sometimes you see, like “Oh shit, there was this World Malaria day, and it was yesterday… I could have had a post about it.” But if someone who is doing it professionally has like a calendar, and warns you about that in the future… Maybe you can post about that and suggest people do this, or whatever… because I notice that it actually works out really well.”

“It would be cool to have some kind of slide or something to show people the overarching organisations, who they’re in touch with, how they’re funded… which are recommended charities… which are charity navigators etc. We could make one ourselves, but if one existed already, that would be cool.”

“Something like a list of all those things and then a link to examples of how other clubs have used it.”

Finally, interest was expressed in the conversion of existing materials into different formats, such as video or audio.

“People have different preferences for consuming information. So maybe if you took the stuff that’s been… like I really love to read things but I know that some people love podcasts and audio. I love video as well. I probably like video better. So maybe you could refocus stuff that people really enjoy into other mediums.”

Websites and Tech Support

In reflecting on their experience of tech resources and support, respondents referred to group websites, paid Meetup.com subscriptions and the EA Hub. We did not receive any feedback about the EA groups platform launched by CEA.

Websites

There is some variety in respondents’ attitudes towards websites. A vocal minority rated website provision highly helpful and impactful. Some respondents had not yet started using a website, and were keen to arrange one in the future.

“We did have a seperate website, but it was terrible. I was like ‘Oh dear God…’; we were extremely grateful. Super helpful! We could have done it ourselves but it would have been ugly and awful and gross.”

“The website I’d probably rate as 9 or 10 (in terms of usefulness). That thing is just such a great go to.”

“A few months ago I wanted to create a website and LEAN offered to help. In the end I didn’t have time to do so, and now I’m wondering if there’s something that is already ready to set up, where I just have to put the text in.”

“We asked [LEAN] for a domain and [LEAN] brought up a webpage for us, which was really great. I think that a new group that is growing – maybe not on the first day or first month of activity – at some point all group should have websites, and if we can do that at a low cost then that’s great. We use that, and especially the little stuff that is concerned, like email aliases so that I can send an email as someone who is behind an organisation, not just a gmail address. And I think that really does make an impression – especially for organisations you might come into cooperation with.”

Among respondents who seemed lukewarm about their websites, most framed the problem in terms of a lack of time or technical skill with which to customise their sites and exploit them to the full.

“I guess it would be nice to revamp our website, which isn’t very high quality. So one thing would be having a web developer who could help us with that – that’s great. Emails would be good, and webhosting would be really nice because we don’t have anyone who maintains our website actively. So it’s very bare bones, and isn’t that nice right now.”

“I’ve been meaning to do a website, mailing list, post on social media etc. but I don’t have time.”

“A good website would be nice. We have a lot of knowledge I guess, but we’re not good at website. So yeh… the website is the biggest help.”

However, one respondent was explicitly negative about the value of websites for groups.

“I think many Universities operate through mailing lists and the University pages, and though the website… a few people have used it, I find we’re not making use of it. So they have been useful but I think you’d have diminishing returns if you invest in those because I don’t find many groups using them.”

Meetup.com

Meetup.com is a social media platform tailored to event scheduling and group promotion and management. Group members can access the site and join groups for free, but creating and administering a group on Meetup.com requires a paid subscription. Currently, LEAN covers Meetup.com subscription for group organisers on a case-by-case basis.

A similar pattern was true of Meetup.com to that which emerged from the website feedback. Namely, a minority of organisers find the platform enormously useful, while others find it modestly useful. We did not receive any negative feedback about Meetup.com at interview. The majority of EA groups do not have a Meetup.com group.

“Given my goal, which is meeting other Effective Altruists, I think Meetup.com has been pretty successful. Like we’ve got a very decent showing every week, and new people I didn’t know of before. So that’s been good. Good but not amazing.”

“The Meetup group I’d say is 4 out of 10 (in terms of usefulness). We’ve attracted a couple of people. I like the fact that people don’t have to have Facebook. It feels more inclusive even if we don’t glean people from it, although we actually did.”

“…in Germany a lot of people don’t use Facebook, because privacy is something people really value here. So it’s sort of tricky to reach people who don’t have Facebook.”

As illustrated in these examples, there are concrete cases of counterfactual membership gains as a result of using Meetup.com, including one group which is solely maintained through Meetup.com. Organisers also value the platform for enabling them to reach individuals who do not use Facebook.

The EA Hub

www.eahub.org is a website created by Rethink Charity (formerly known as .impact) as a community resource. The EA Hub was initially put together using submissions from the first EA Survey. The site offers a donations registry which encourages EAs to publicly list donations in order to motivate others. It also offers a list of personal EA profiles where EAs share basic personal information such as preferred cause areas, career plans, and so forth. LEAN has created public profiles for EA Groups on the Hub since its’ inception. The site offers a map of EAs and a map of EA groups designed to help people to access a quick visual of nearby like-minded groups or individuals. Finally, LEAN uses the domain @eahub.org to provide group organisers with official email aliases in instances where a group does not have a website. (Where groups do have a website, we provide group specific domains which are then for their email aliases).

While not many organisers mentioned the EA Hub, one organiser found it to be instrumentally influential in the formation of his initial committee. Another found the map useful, and the third felt that the Hub had once offered potential, but had become outdated and redundant over time.

“My EA Hub profile is how the first two [group members] found me.”

“I really like the map [of EAs]. Looking at the map and seeing where groups are is really cool.”

“I really like your efforts. I like the EA Hub!”

“The EA Hub was a great initiative that sort of feels redundant now. It’s not really working, to be honest. It doesn’t meet people’s expectations of how software should work in 2017 – I’ve received that feedback on a couple of occasions. And some of the functionality is broken.”

Connecting and Introducing

While there is no explicit service for introducing specific organisers to one another, LEAN has historically helped many organisers on a case-by-case basis, introducing them to other EAs with compatible needs or interests, or similar locations [3]. Respondents raised this as an example of external support that was valuable and impactful.

“If it weren’t for LEAN and Rethink Charity, I would be even further behind when it comes to actually finding EAs on the ground in [Country]. People who self identify as EA in [Country] are very, very rare. I can pick out less than ten right now… the people who self-identify as Effective Altruists, as far as I know of… that I’ve engaged with on a one-to-one basis. I would be starting with nothing, basically, if it weren’t for you guys. I think that’s something, maybe hard to systematise and predict, but… those early introductions that help you build critical mass… you’ve been really helpful with that.”

“Making sure that other group and project email addresses are up to date is so helpful.”

“Rationalist groups are easy to find, but something harder to search for is if there are EA corporations or non-profits in [City]. I was surprised to learn that [EA Organisation] was based here.”

“[LEAN] has been a great support for me. Almost from when we met there was a consistent line of communication, and I would get a lot of advice and he would always connect me to the right people, which really worked out.”

“You know I hadn’t even thought of trying [to contact other organisers], so after this conversation one of the things I’m definitely going to do is just reach out to the guys in [a place LEAN recommended the organiser contact] and see what exactly they’re doing regarding such a challenging regulatory environment.”

“‘I had help from [LEAN] in the beginning when she connected me to some people… So far [she] has been super helpful, because if I didn’t have [her] I wouldn’t have two members who are already in EA, and they’ve put a lot of work into it. So I guess that’s…very precious stuff.”

One-to-One Support

This section refers to input and feedback that organisers receive from LEAN and other organisations. In the case of LEAN this involves personal discussion between an organiser and a member of the LEAN staff, but it may also involve participation in the mentoring programme.

Respondents who mentioned having drawn on one-to-one support found it useful. Furthermore, although many respondents had not formally drawn on this support from LEAN or any other EA organisation, many mentioned positive interactions with independent EAs or other organisers. Respondents were widely enthusiastic about the value in continuing to make such opportunities available.

“I chatted with [LEAN] and, that was actually a really good chat – I still have the notes from that – uh, this was still in the mode when I was the only real organiser, and I just wasn’t able to act on a lot of that stuff because I didn’t have the motivation, I didn’t have the bandwidth to be like, to feel like I was the only person pushing the group along.”

“More frequent conversations and more networking would be highly helpful, like keeping in touch and finding out what other groups are doing, and picking their brains… picking their information. That would be very helpful, I think. So right now it doesn’t seem like I have any of that.”

Nevertheless, some respondents hadn’t been aware that support was on offer from LEAN, or they had started up before it was available.

“The part that was the most frustrating was when we started we had nothing. People only start offering help after the first year. But if we had people advising us as soon as we started, it would have cleared things a lot. So I really think it’s very important that as soon as you hear that an EA club anywhere in the world has started. Like… very fast… run to them and offer support. That was probably the number one thing. ‘Cos things are so much easier now, and they could have been at the start if we had that support at the start…”

Other respondents felt that their needs were too specific or unique for general support to be helpful. For example, one organiser was most in need of input from an Islamic religious expert, while another stressed the significance of cultural and language barriers.

“A lot of these things is like…I mentioned before that there is this language barrier and cultural difference. I think these are not an easy thing to get bridged.”

One or two organisers mentioned that they had considered asking for support but had found themselves too busy to get around to it.

Funding

The most frequently mentioned need among organisers was for improved financial support. Funding was also the category of external support that groups found most helpful when asked to reflect on their development.

“I would say the hardest part was, uh, not having constant funding. We were pretty limited in what kind of events, or what have you, we’d be able to put up. I mean I sunk something like $200 into like the initial advertising of the club that was never made back. I don’t expect it to ever be made back. But of course it wasn’t just me… everyone kind of helped contribute to it.”

“…he called the group to give me some money. So we decided to print some posters, some leaflets… to invest in Facebook advertising – as I remember. And we got thirty people attending. It was like… that was a nice push right there, because these guys were really into Effective Altruism. They were really interested, they knew what they came for. And out of those guys, I think two of those guys are still with us and are key people for the organisation. Or even more…like three to five out of thirty. And that was three years ago!”

“Getting the Meetup group set up was helpful mainly from a monetary perspective, honestly. I personally don’t have much money so that was the main thing.”

Funding is especially crucial for groups that want to level up their operations, for example transitioning from a local to a regional group, or transforming into a foundation or charity.

“How to have enough resources to focus on this stuff. We are students, we work, we do all kinds of stuff, so how can we also do Effective Altruism [Country]? And this has been a huge challenge with regards to money, with regards to time, with regards to all kinds of resources.”

“If we really want this club to shine, we will need someone to work on it that isn’t also a student. I’m not sure exactly how this would work, or what bureaucratic limitations that person would have, but we need a more dedicated team than we currently have.”

“I no longer think of us as a local chapter because we are becoming an organisation. And that makes it so that you need to, you know, grow… to meet the challenges that we are now facing.”

“There are all these small NGOs that get money to go and talk in schools about recycling or racism, and all of these small organisations found money for this to happen, so it should be possible for Effective Altruism to do so.”

In most of these cases, the need for financial resources is connected to the need to hire paid, dedicated staff in order to realise the opportunities and goals that groups have.

Venues

Another frequent challenge for organisers lies in securing appropriate locations for regular meetings. In some cities, the challenge lies in finding places that are conveniently accessible to members from different, far flung locations. Others know of suitable locations but can’t afford the booking fees, or have been struggling in crowded and noisy restaurants, cafes or bars.

“What are good venues to do meetups? Here we have a problem because venues are bad somehow. Like we have to pay or something else, you know?”

“Currently we have a bit of a problem with a place we’re doing the meetups because we began in a restaurant, but it was too small. When it’s loud, you can’t really talk. Especially for social meetups. Because we don’t have student union membership, we can’t book a University room. We have another room, but it’s quite expensive.”

“The organisation I’m at right now, they’ve helped us by just offering workspace. We can organise events here at cost price and have our meetings handled, and things like that.”

“The main challenges have been booking rooms and working out what events we could run.”

Speakers

Organisers also frequently mention access to EA speakers as a significant resource. Either groups struggle to find speakers, or they are based in cities or Universities with a good concentration of speakers and reflect on this as having been instrumental to their success.

“Another thing I’d love to be able to do is find guest speakers from around the world and help pay for them to get here, so they could give guest lectures.”

“For the launch event we really didn’t want to do it ourselves. We really wanted an external speaker who could attract a few more people because if we just do it, it’s going to be our friends there. And so we asked them (a regional EA organisation) if any of them could come and give a talk and they said ‘no we don’t have any time in those months’.”

“I guess maybe for prominent EA speakers. People like Peter Singer and whoever.

“Also just knowing which EAs are where. Like when EAs are visiting a city… I can imagine a really awesome EA coming to town and you just not knowing about it.”

“Is there any information on what speakers are around in what areas? It’s hard to keep tabs on certain people, and if they’re coming through to [City] it would be good to say ‘Hey! While you’re here, come to our group!’”

Group Calls

A few organisers mentioned group calls as having been inspiring and helpful. Some organisers had participated in calls organised by LEAN or CEA, while others had arranged these calls independently.

“In general I think that being able to have skype conversations, and using online tools has helped a lot. I can imagine that it multiplies.”

“I think it’s very valuable to know the broader EA landscape. So I think that like… both CEA and LEAN – you’ve been cooperating on that. The talks, the phone calls or the Skype calls… they’re really valuable. Just talking to other people, that’s not easy to do just as a stand-alone EA group.”

“I haven’t had much direct contact. I would love to be active in any group calls if you share that with me.”

The reasons for valuing these calls were similar to those for valuing personal connection with the wider EA community. Although feedback for the calls was mostly positive, two organisers mentioned the fact that group calls had limited value for their needs given significant differences between the opportunities and challenges they were facing and those that the majority of call participants were discussing.

“They were kind of useful but there were so many different kinds of groups. How to help your group at the stage it’s in? Sometimes I’d have a chat and they were like “Oh we’re doing all these things” and I was like “We can’t do those things… we’re not a University, we’re just little”…It was difficult.”

“I’m also aware that with a lot of groups… with student groups and younger people, less experienced… I understand that that’s a priority. Um so no, I think it’s clear for us that ‘No, that’s not for us. That’s more for student groups, but that’s ok’.”

Conclusion

Having summarised all of the empirical data from our quantitative and qualitative research, the penultimate report will synthesise both sources in order to develop a thorough interpretation of these findings and their significance for the LEAN project, and for EA movement building generally.

Acknowledgements

This report was written by Richenda Herzig with editorial contributions by David Moss and Tee Barnett. Interview recruitment was overseen by David Vatousios. The interview sample was divided evenly between the interviewers (Richenda Herzig and David Vatousios). Interview recordings were transcribed by Richenda Herzig, David Vatousios, Brooke Jackson, Avi Iyer, Kaitlin Alcantara, Sharmin Tuli, Leticia Pena and Ashley Francis.

We are highly grateful to Greg Lewis for his input as an external advisor.

We’d like to offer our deep gratitude to each EA organiser that agreed to participate in the interviews, both for the trust and generosity in sharing your thoughts and experiences, and the time sacrificed.

Endnotes

[1] The impact assessment utilises a mixed method social research strategy, including both quantitative and qualitative components. In this report, we present transcribed interview data with descriptive commentary only, leaving full strategic analysis for a later article. Report #4 will describe our methodology in full.

To briefly recap, we selected a large sample of EA groups based on length of time running, and a desire to include a mix of different nationalities, different group types (University and local), and different affiliations (e.g. groups mostly served by CEA, LEAN, EAF or TLYCS). Of this sample, we interviewed each organiser that was willing to participate, which whittled our selection down to 31 organisers (from around 70).

Our interviews were conducted over skype with the exception of one which took place face to face in Vancouver. An audio recording of interviews was taken on our mobile phones in the early stages. Eventually we invested in commercial Skype recording software, which produced full video recordings. The first two interviews were jointly conducted by David Vatousios and Richenda Herzig for training and calibration purposes. Thereafter the interviews were loosely split between each interviewer for convenience (interviews took place across a range of different time zones, which meant that personal scheduling determined which interviewer was available for any given interview) and in order to improve our objectivity.

Recordings were transcribed by Richenda Herzig, David Vatousios and a team of four remote LEAN volunteers (who each signed non disclosure agreements).

In summarising findings from this data set, we paid attention to the frequency that a theme would be mentioned by respondents, the weighting respondents gave to these themes as expressed by length of time and volume of detail used in discussing the theme, as well as their explicit evaluative comments. Pragmatic and clarity considerations also affected which extracts could be used. For example, there was considerable overlap in terms of the themes that would be covered in data units as small as one or two sentences. This meant that the number of quotes where an issue was clearly isolated was small. For less isolated quotes, much larger extracts would have been necessary to include in order to render the extract clear. Finally, inconsistency in the quality of transcription meant that some data could not be appropriately formatted and included within our time frame.

[2] Each example included has been anonymised in order to protect the privacy of respondents and their groups. Where personal identifiers could not be removed, pseudonyms were used. Where changes were made to quotes, either for anonymity or for clarification, square brackets were used.

[3] We assume the same is true of CEA and EAF.

Share post:

2017 LEAN Impact Assessment: Quantitative Findings

2018-01-05 General

The Local Effective Altruism Network (LEAN) is a Rethink Charity project initiated in 2015, which focuses on providing material and informational assistance to university and local EA groups around the world.

This document is the first in the LEAN Impact Assessment Series, summarising relevant data from the 2017 Local Group Survey, which is used to assess the EA local group network and the effectiveness of LEAN’s services.

The assessment utilises a mixed method social research strategy, including both quantitative and qualitative components. Our quantitative research relies upon relevant [1] data from the 2017 Local Group Survey [2], which was conducted in collaboration with the Effective Altruism Foundation (EAF) and the Centre for Effective Altruism (CEA) mid-2017. Our qualitative research is made up of over 30 semi-structured interviews that LEAN held by video call with group organisers around the world, ranging from 20 to 40 minutes in duration.

Findings are divided into two overarching categories: “EA Groups” and “Support and Resources”. The first includes data related to the scale, nature and impact of groups, (e.g. membership numbers or funds raised). The second includes data on particular group support strategies, and their popularity and impact.

In this report, and in the qualitative finding summary to follow, we offer descriptive commentary only, leaving full strategic analysis and interpretation for a later article.

Table of Contents

EA Groups

Support and Resources

Personal Feedback
Practical Support and New Ideas
Group Video Calls
Written Guides
Websites and Technical Support
Premium Meetup.com Subscription
Local Group Newsletter
Group Organisers’ Mentoring Programme
Group Organisers’ Facebook Community
EA Groups Slack Team

Conclusion
Acknowledgements

Survey Sample

The 2017 Local Group Survey was sent by LEAN staff to every EA group organiser on record [3] via email and Facebook message. The survey was also posted in EA Facebook groups, and advertised in the EA Newsletter and the Local Groups Newsletter.

Although 374 entries were submitted to the survey (138 of which identified as group leaders and 236 as group members), 292 entries remained once the data was cleaned [4]. Among these, 98 identified as organisers, and 194 as members.

The survey was split into sections containing questions reserved for organisers, and questions open to organisers and members. Where groups had more than one organiser, they were asked to nominate one organiser to complete the questions designated for organisers. Surplus organisers completed the survey as members. Organisers answered on behalf of their groups for the organiser questions, and on behalf of themselves for the member questions.

Some questions attracted many more responses than others, and several participants chose to skip certain questions. Throughout this report, response levels are indicated as fractions of the total respondent category, in order to signal this difference. For example, if there are 75 responses to a question restricted to organisers, this is displayed as 75/98 where 98 is the total number of organisers that completed the survey.

Our methodology is explained in more detail in a forthcoming post in this series.

EA Groups

Group Demographics

Number of Groups and Group Size


Organisers reported an average of 50 group members, and a median of 10. The data are partially determined by the different criteria respondents used for defining membership. For instance, some used the size of their Facebook or Meetup groups, while others counted only individuals who had attended group activities on a regular basis. Based on these numbers though, slightly more than 78% of members are within the top 10% of groups by size.

Group Type


56 participating organisers were from local groups, and 37 organisers were from University groups. Of course, many members of local city based groups may still be university students. Some groups may be better understood as hybrids between the two categories. It is also possible that as the EA community is aging, EAs who joined the movement during university are progressing into local non-university groups.

Group Leader Succession

As a rough indicator of the stability of groups, we asked organisers to estimate the likeliness that their groups would continue were the current organisers to step down.

The results suggest a degree of vulnerability, and dependence on particular organisers.

EA Activities of Organisers and Members

We asked members and organisers whether or not they had ever engaged in the following activities:


We supplied no participation count for this question because respondents were only given the option to add a mark if they engaged in the relevant activity. Therefore, a lack of marks from a respondent could mean either a lack of engaging in the question, or it could mean that the respondent doesn’t engage in any of the activities. Any participation count would therefore not be informative.

An additional limitation of these results is the fact that categories such as “volunteered at an EA organisation” were not sufficiently defined. For example, some respondents interpreted time spent organising their groups as voluntary work for an EA organisation, while others did not.


In an open ended addendum to the question, respondents reported additional ways of investing time in EA:

  • Thinking about EA
  • Direct EA work
  • Producing EA content
  • Researching EA
  • EA informed career transition
  • Applying for EA related grants
  • Pitching EA to individuals
  • EA aligned policy work
  • Earning to give

This suggests that members (and organisers) of EA groups are engaged in Effective Altruist activities and the movement more broadly. Almost by definition, group membership involves social interaction with other EAs. It is clear, however, that this is just one of many activities that members are involved in.

Commitment and Lifestyle Changes

Increasing Engagement with EA

Perhaps the most important success criteria for EA groups is their ability to attract people to Effective Altruism, and to retain their interest and commitment. We included the following questions in the Local Group Survey in order to gauge this:

The following table and graph summarise data on how many respondents considered themselves Effective Altruists prior to attending their first group meeting:


It may appear striking that some organisers did not consider themselves EAs until after their first group meeting. Note that it may be that organisers were converted to EA after attending their first group meeting as a member, but became an organiser after becoming an EA. It may also be the case that some respondents are reluctant to apply the label ‘EA’ to themselves e.g. until they’ve started doing something they see as concrete for EA (such as organising an EA group).

While it is difficult to discern causation from these figures, they do at least confirm that a sizeable number individuals become EAs only after beginning to attend a group. This confirms that groups are not merely reaching people who already identity as EAs and nor are they merely reaching non-EAs who never subsequently come to identify with the movement.

This graph displays the number of people who group organisers estimated attended each group’s events with little or no familiarity with Effective Altruism:

Responses are widely distributed, with a small number of groups reporting very large numbers, but most responses clustered around 30 new event attendees with little familiarity with EA, and with most responses (44/78) falling between 5 and 50. It is important to see this in the context of group size, with all but 10 of the groups who responded to this question had <50 group members. The ratio between group members and total event attendees who were unfamiliar with EA varied widely, between 1:0.375 (a group with 40 members and 15 new event attendees) and 1:43 (a group with 14 members and 600 event attendees who were unfamiliar with EA).

How much of a factor was group involvement for engagement with EA?

When asked how much of a factor group involvement was for their engagement with EA, respondents gave the following answers:

Most members (102/178) and organisers (45/72) report involvement with their EA group to be ‘large’ or ‘very large’ factor for their engagement with EA, with the remainder of the rest being ‘moderate’ responses.

Number of members becoming ‘actively committed’ to EA due to groups

We also asked for counterfactual estimates of the number of members who became ‘actively committed’ to EA (for example, lifestyle changes, direct action or donating money to effective causes) as a result of involvement in groups.

The median number of estimated counterfactual active commitments is 5 and the majority (43/63) fall between 1 and 10. A small number of groups report substantially higher numbers (e.g. 50-100 counterfactual commitments). The top 10/63 groups account for slightly more than half the reported commitments (352/617.5).

Looking at this in relation to ‘group size’ finds a positive correlation between the number of members a group has and its number of reported active commitments.

Thought and Behavior Change Since Being Involved with a Local Group

We asked all respondents whether their world views or behaviours had changed since becoming involved in an EA group.


We also asked them to indicate whether or not any changes reported were likely to be impactful.

A substantial majority of members and organisers alike report that the way they think about the world and behave has changed since being involved with a local group and that they expect to have more social impact as a result of this change. This does not necessarily suggest causation between the local group and their increased engagement and efficacy.

How many of your current members do you expect to choose careers based on EA recommendations or thinking?



Most groups report between 1 and 5 members choosing careers based on EA principles (median 4). The mean (7.29) is dragged upwards by a small number of groups with much higher (up to 50) numbers of career choices based on EA.

A natural question to ask is how this relates to group size. Are the largest groups simply accounting for many more of these outcomes (due to their much greater size)? The first graph shown here would seem to suggest this, with the group with the largest number of EA career choices by some way, also being the largest, and all but one of the groups with the highest number of EA career choices having >100 members.

It is a further question, however, whether the largest groups are better at making conversions (e.g. getting members to make EA career choices). The graph below would not support this conclusion. We see here only a weak correlation, but it might appear that the largest groups (responsible for the most EA career choices) in absolute terms, have a relatively lower % of members making EA career choices. This does not seem sufficient to suggest that larger EA groups are worse at making conversions however. A plausible explanation might be that smaller groups contain a disproportionate number of dedicated EAs (for example, a small group with 5 members might contain two EAs sufficiently dedicated to found and run a group), compared to the largest groups which may have many hundreds of new members.

If applicable, how much of a factor are or were EA principles in planning your career?



A majority of members and organisers alike report that EA principles were a large or very large factor in planning their careers. Notably, though perhaps unsurprisingly, organisers disproportionately indicated that they were a “very large” factor in planning their careers, whereas among members there were relatively more moderate and large responses.

Examples of Notable Group Members Becoming Active EAs

Organisers responded to the question: “Please name any current or past group members who have gone on to become active in the wider EA community. This would include going on to work at an EA organization, starting an EA project, becoming a thought leader in the movement, earning to give, and/or representing EA in other public ways.” We refer to these individuals as ‘hits’ who initiated or increased their level of involvement in EA after group involvement.

In total 121 ‘hits’ were reported in this open comment question. Note that low response rates may obscure the number of group organisers who would report 0 ‘hits.’

How valuable do you find your group’s activities?


As this graph shows, majorities of both organisers and members rate their group’s activities as valuable or very valuable. Notably, members appear strikingly more positive than organisers, “very valuable” being their most frequent response by some way (80), followed by valuable, with 135 out of 144 selecting these two options, whereas organisers’ responses are centred around valuable (30) and moderately valuable (22).

Funds moved

Group organisers provided estimates of the money raised through group fundraising activities, the money raised through the private donations of members, and the counterfactual GWWC pledges raised by groups.


While substantial funds have been collectively raised by groups, the majority of the funds come from a small number of groups.

Nevertheless an appreciable number of groups have fundraised significant (i.e. $100 to $1000) amounts, as seen below (note the log scale on the y axis).

Note that we would expect that non-response would be higher for groups who have not run fundraisers or who raised very little, so there may be a longer ‘tail’ of groups raising $0.

This table summarises group organisers’ estimates of the private donations made by group members who counterfactually would not have donated (but for group involvement):

Total estimated counterfactual private donations are dominated by a small number of groups reporting very high figures (note the log scale on the x axis). However, a substantial number of groups are reporting significant sums being donated. An important caveat is that respondents may have over-estimated some of the figures for various reasons (for example, including future donations).

Note, as above, that non-response rates may conceal a higher number of groups who would estimate very low donations.

Finally, organisers provided estimates of the counterfactual Giving What We Can pledges secured through their groups.


Most organisers report few counterfactual pledges (pledges which would not have been taken without the influence of the group), with most reporting between 1 and 5. Indeed, the vast majority of responses fall within 1 and 11, while 2 groups report 40 and 75 counterfactual pledges respectively, and 16 report 0 counterfactual pledges.

Overall, these data on funds are speculative, and should be treated as such. However, it appears that groups have a non-trivial role in the movement of funds to effective causes.

LEAN Support and Resources

In this part of the report we summarise evidence regarding the usefulness of services which LEAN provides in assisting the operation of groups [5]. More data on groups’ experiences of outside support will be shared in the qualitative report.

General feedback on LEAN and other EA organisations

We asked organisers:

“What outside help has been the most useful to the operation of your group?” (Respondents could select multiple options.)

“Other” was made up of specific University student unions, larger EA groups in similar regions (EA NTNU, EA London and EA ANU), value aligned local organisations, and individual EAs.

CEA (39) was the organisation most often selected as ‘most useful’ to the operation of groups by organisers. Note that respondents could select multiple organisations if they wished. LEAN (22) and EAF (22) were joint second most commonly selected organisations. TLYCS was selected 11 times, and remaining options were each selected 2 or fewer times. CEA-affiliated 80,000 Hours (21) and Giving What We Can (13) were selected separately by some respondents. A favourable bias towards LEAN is possible given the fact that LEAN distributed the survey.

Feedback on Specific Services and Resources

Personal Feedback

LEAN offers organisers personal support, on demand, via video call, social media and email. We asked organisers: “In your opinion, how useful is personal feedback and support via social media, email and video call?”


A clear majority report that personal feedback and support of this kind is useful or very useful.

Practical support and new ideas

We asked organisers: “In your opinion, how useful is it to receive practical support and new ideas for group activities?”

Practical support and new ideas for group activities are generally rated as useful or very useful (75/80), with only (3/8) finding them either not useful or not at all useful.

Video Calls

LEAN hosts video calls to help share best practices between groups. Organisers responded to the question: “In your opinion, how useful is it to host video calls about group management topics?”

A majority of organisers reported video calls about group management to be either useful or very useful.

Written Guides

LEAN is among many EA individuals and organisations to have produced written content for EA organisers. We asked organisers: “In your opinion, how useful are written guides (with a focus on practical and strategic aspects of organising groups)?”


A clear majority of respondents (65/78) considered written guides to be useful or very useful with only 1 respondent out of 78 offering a negative rating.

Websites and Technical Support

LEAN provides hosting, domains and basic content management for over fifty EA group websites. Organisers were asked: “If your group uses a website, do you believe that it makes a non-trivial difference in the effectiveness of your group’s outreach efforts?”

While a majority of the groups who used (group) websites find them significantly useful, a notable minority find them no more than trivially useful.

In addition, we asked: “In your opinion, how useful is technical support (for instance, subscriptions for online services, free websites, group email addresses)?”

A majority (52/73) of respondents report that technical support of this nature is either useful or very useful, compared to 13 and 8 groups being neutral or not finding it useful, respectively.

Premium Meetup.com subscription

LEAN provides free Meetup.com accounts for interested organizers. Organisers were asked: “If your group uses Meetup.com, please give an estimation of the % more attendees you have attracted as a result of using the platform in addition to – or instead of – alternatives?”


*It should be noted that most EA Groups don’t use Meetup.com and would not have been able to answer this question.

While many groups gained modest increases in members from using meetup.com (median 15%, mean 21.42%), a small number gained very significant increases.

Local Group Newsletter

LEAN leads a regular newsletter for EA groups with support from EAF and CEA. We asked organisers: “In your opinion, how useful is the Local Group Newsletter?”

Many more respondents rated the local groups newsletter (N.B. not the EA Newsletter) as useful or very useful, (32) than not useful or not at all useful (34), though many were neutral (24).

These results should be contextualised, however, by responses at the end of the survey which asked whether respondents wished to be added to the local organiser newsletter:

This shows that 49 organisers or more have not received the newsletter, which limits the usefulness of the earlier responses.

Group Organisers’ Mentoring Programme

With support from CEA, LEAN launched a mentoring trial programme, connecting experienced organisers with new ones from August 2017. We asked organisers: “In your opinion, how useful is the EA Organiser’s Mentoring Programme?”

The majority (32) of respondents found this program neither useful nor useless, with 16 finding it useful or very useful and 4 finding it not useful. These results may indicate that the majority of organisers are simply unfamiliar with the program due to its recent release.

EA Organisers’ Facebook Community

LEAN supports a Facebook group for EA Organisers in collaboration with EAF and CEA.
We asked organisers: “In your opinion, how useful is the Facebook community of group organisers?”

A decisive majority of organisers found the Facebook group to be useful or very useful.

EA Groups Slack Team

LEAN supports a Slack channel for EA Groups in collaboration with EAF and CEA. Organisers were asked: “In your opinion, how useful is the EA Groups Slack Team?”

Slightly more organisers found the Slack team to be useless (13) rather than useful (10), with the majority (34) being neutral.

Conclusion

Evaluating the impact of LEAN and the strategic implications of these results will be deferred until the LEAN Assessment Strategy report, which will follow in this series of articles. We will also draw on the qualitative data we have gathered in a separate report to help interpret these findings.

Acknowledgements

This report was written by Richenda Herzig. David Moss, Peter Hurford and Richenda Herzig analysed the 2017 Local Group Survey data. Editorial input was provided by Peter Hurford and Tee Barnett. Thanks to Ellen McGeoch for assisting in survey design and formatting for the 2017 Local Group Survey. Thanks to David Vatousious for distributing the survey across the network and for recruiting participants. Thanks also to Kaitlin Alcantara for data entry and filtering.

We are highly grateful to Greg Lewis for his input as an external advisor.

We would also like to express our thanks to Harri Besceli from the Centre for Effective Altruism (CEA) and Jonas Vollmer from the Effective Altruism Foundation (EAF), who collaborated in writing the 2017 Local Group Survey. We are grateful to CEA for generously supplying free EA t-shirts to respondents.

Last but not least, a big thank you to all organisers and members who took and shared the survey!

Endnotes

[1] The LEAN Impact Assessment is distinct from the 2017 Local Group Survey. While the survey results supply a substantive base for the assessment, the survey was a collaborative project between the Centre for Effective Altruism, the Effective Altruism Foundation, and The Local Effective Altruism Network (LEAN). Findings from the survey that were not relevant to this assessment may be shared at a later date.

[2] Due to the number of personal identifiers in the data set, it is not possible at this point in time to make the raw survey results publicly available. At a later date it may be possible to release partial anonymised findings.

[3]LEAN collaborates with CEA and EAF to maintain up to date, comprehensive records of EA groups and their organisers.

[4] Entries were deleted if they were blank, or sufficiently incomplete as to render the submitted data useless. Other deletions included garbled or illegible responses and duplicates.

[5] Of the support categories included in this section, some have historically been provided only by LEAN, whereas others have been provided by various individuals and organisations in EA.

Share post: