The approach

Approach to evidence-based conservation learning

Explore the five practical steps for evidence-based learning in conservation.

In this initiative, we use evidence to test critical assumptions in conservation practice. We have elaborated five practical steps for using assumptions and evidence to answer essential learning questions.

The approach we present here is a first attempt at formulating a method for evidence-based learning. We are continuously improving our process based on its application in practice and input from the conservation community.

By following this approach and optimising evidence use, it is possible to unveil crucial lessons about conservation strategies.

Watch the webinar recording

Learn about the need for using evidence in conservation, our practical five-step approach, and the key insights on the topic of capacity-building.

Watch the recording on YouTube.

Thumbnail of a slide from the webinar, listing the speakers, with the YouTube play icon on top.

Five steps for evidence-based learning

The core elements of evidence-based learning are: a well-defined learning topic corresponding to a conservation action, relevant learning questions that help to learn about the mechanics of the action, and explicit assumptions to test with evidence. The five steps for evidence-based learning focus on and utilise these elements: 

  1. Define the overall learning topic
  2. Develop learning questions and formulate assumptions for each
  3. Collect evidence related to the assumptions
  4. Assess the evidence to determine whether it supports the assumptions
  5. Compile and conclude what we have learned about the question
The approach to evidence-based learning in conservation. This is a first attempt at formulating the method and we are continuously improving our process.

Background

Already in the early 2000s, the conservation community identified the need to use evidence for decision-making to increase the effectiveness of conservation strategies11,12,19. Since then, the efforts to establish an evidence-based conservation practice have been twofold: 

  1. Published evidence: Approaches have emerged to collect and compile existing published evidence18 and to make it accessible for conservation practitioners through online libraries and databases. Examples of these efforts are the CEEDER database of the Collaboration for Environmental Evidence (CEE) and Conservation Evidence at the University of Cambridge.
  2. Evidence from practice: The conservation sector became more rigorous in project and programme design and management. The aim was to improve conservation practice through adaptive management13 and standard practices3,4,5,7.

Building on these developments, the conservation community has recently developed concepts for defining and using evidence in practice8,15,16 and proposed an approach to assess practical evidence for specific conservation strategies17. 

The five-step approach we are presenting here heavily builds on these developments over the last 20 years.

Step 1 – Define the learning topic

Define the topic for learning, why learning about this topic is essential, and who is (likely) interested in the lessons learned. This step forms the context for the other steps of the process. 

Our approach to defining the learning topic

Our first challenge was to select a handful of learning topics from the MAVA portfolio of conservation grants. The portfolio spans roughly 1700 grants, addressing numerous conservation issues across different geographic regions and nearly 30 years. 

Potential learning topics: We started with a quick scan of all 1700 grants. For each grant, we identified what main conservation strategies it covered. We then cross-referenced these strategies using a standard classification scheme of conservation actions14,6. This helped us list roughly 100 different conservation strategies implemented through the MAVA portfolio.

Potential evidence: We then narrowed the list down to 14 conservation strategies for which we expected evidence to be available. For this, we looked at the total number of grants that included the strategy and the total financial investment by MAVA in those grants. We assumed that the higher the total number of grants and the higher the total financial investment – the more likely it would be that we would find enough evidence. 

Relevance and interest: Finally, we ranked each of the remaining 14 strategies by evaluating how relevant the learning topic is to the broader conservation community. We combined this ranking with our interest in each of them as learning topics.

Through this process, we arrived at four selected learning topics:

  1. Building capacity of key stakeholders and conservation organisations
  2. Establishing strategic conservation partnerships and alliances
  3. Providing flexible conservation funding
  4. Basic conservation research and monitoring

Step 2 – Develop learning questions

Find focus in your learning topics by formulating relevant learning questions and associated assumptions that you can test with evidence. You can build the necessary framework for selecting learning questions by creating a theory of change that captures the pathway from conservation action to desired outcomes and impact. This step helps you to focus on desired learning outcomes.

Our approach to developing learning questions

For each of the four learning topics, we developed a theory of change icon-question. We used a results chain diagram9 to explain how we assumed the conservation action would lead to specific results. Whenever possible, we started with a generic results chain from the Conservation Actions & Measures Library (CAML). We then refined this generic chain through various rounds of reviewing and validation to finally arrive at a version that best represents the work implemented through MAVA-funded grants.

We used the results and the assumed causal relationships between results to tease out essential learning questions icon-question. We then formulated one or more assumptions icon-question for each learning question to test with evidence.

The theory of change for the learning topic Providing flexible funding. Note that the darker purple boxes contain the learning questions. The light purple boxes show the associated assumptions. Click the image to see it in full size. For more details about this learning topic, see the page for Flexible funding.

Step 3 – Collect Evidence

Once you have defined the learning questions and related assumptions, you can start collecting evidence. Evidence can come from different sources. Collecting evidence in an ordered and systematic way makes assessing it (Step 4) more manageable. 

Our approach to collecting evidence

We collected all the relevant evidence icon-question for each assumption in the MAVA domain. We complemented this with evidence collected from outside the MAVA domain.

Looking beyond the MAVA domain helped us widen the evidence base and look for generic lessons learned. The amount of evidence we collected from outside the MAVA domain was limited by the available time, i.e. we did not do exhaustive reviews of all published evidence. 

We provide a detailed description of the evidence base for all learning topics in the Learning section.

Sources inside the MAVA domain:

  • Questionnaires to MAVA grantees
  • Discussions and focus groups with the MAVA team
  • MAVA grantee reports, proposals, and evaluations

Sources outside the MAVA domain:

  • Conservation Evidence database through systematic searches 
  • Google and Google Scholar through exploratory searches
  • Discussions with key researchers to highlight additional sources

The different types of evidence we found: 

  • Responses to questionnaires – both qualitative and quantitative
  • Processed financial data
  • Quotes and extracts from reports and articles
  • Results and findings from research articles
  • Conclusions from discussions with expert groups

Step 4 – Assess Evidence

When assessing evidence, two variables play a role: the degree to which evidence supports the assumption and the weight of the evidence. Both of these variables entail careful examination of each piece of evidence. The weighted pieces of evidence together then form the evidence base that you can use to test your assumption (Step 5).

It is essential to maintain consistency in your assessment of reliability and relevance. Group discussions, consultation with experts, and multiple iterations help ensure consistency.

Our approach to assessing evidence

To test our assumptions, we looked at every piece of evidence individually. We assessed the degree to which it supports the assumption and the weight of the evidence. To determine the weight, we looked at the reliability and relevance of the evidence17,2. In formulating this part of the approach, we relied heavily on the work of Salafsky et al., 201915.

Each piece of evidence was assessed on degree of support for the assumption and on weight. Weight is a function of both relevance and reliability.

We had repeated discussions to maintain consistency in our assessments across evidence pieces and learning topics. By working through numerous examples, we established what features are typical for the most vs least reliable evidence pieces and the most vs least relevant ones. Where helpful, we wrote down general descriptors for each level of reliability and relevance to serve as reference material when assessing evidence.

Degree of support

We took a two-step approach to determine the degree of support. 

First, we considered whether a piece of evidence indicates that the assumption is true (i.e. supports it), false (i.e. refutes it), or a mix of both (i.e. shows mixed support). 

Second, if the piece of evidence supports the assumption, how strongly does it do so: does it show some support or strong support? On the negative side, in contrast, it was difficult to distinguish between strength of refutation in this way. 

We therefore decided to judge the support of an individual piece of evidence to a particular assumption as either Strong, Some, Mixed, or Refuting. The degree of support can be illustrated by the colour and symbol of the evidence piece.

To illustrate this, consider the following assumption:

Trained people perform better than untrained people

A study that found improved performance following training would support this assumption. Furthermore, if the vast majority of trainees showed improved performance, support for the assumption would be strong. On the other hand, for a different study showing that only around half of the trainees improved, the support would be mixed. The assumption would be refuted by a study that found either that trained and untrained people perform equally well or that untrained people perform better.

The exact formulation of the assumption greatly influences the degree of support. For example, an alternative assumption to the one above could be There is no difference in performance between trained and untrained people. In this case, the assumption would be refuted by evidence for any difference between the two groups, regardless if it showed trained or untrained people performed better. However, a study showing that they perform equally well would support it.

Weight

To assess the weight of a piece of evidence, we considered its relevance and reliability. 

Relevance refers to whether the evidence piece should be taken into account at all. Evidence pieces with low relevance to the assumption might be considered, but will have lower weight. Similarly, evidence pieces with low reliability also reduce the weight of evidence, regardless of its conclusion. 

We used four weight categories: Very high, High, Medium and Low. The weight can be illustrated by the width of the evidence piece: the wider, the heavier.

The table below shows how we combine relevance and reliability into weight. 

Relevance

We identified two components that together determine the relevance of a piece of evidence: relevance of findings and relevance of context.

Component 1: Relevance of findings

We first assessed how relevant the findings of the evidence piece were to the assumption that we were testing. We judged the relevance of the findings high if we could meaningfully connect the findings to the assumption. We considered the relevance lower if we needed to make an additional assumption. 

To illustrate, consider the assumption Trainees apply their skills in their ongoing work. Imagine that a piece of evidence found that a group of trainees feels highly motivated to use their new skills following a training course, but without recording whether they also use the skills. We could use this evidence to assess the assumption, but if we do, we must also assume that this motivation translates into trainees using the skills. Therefore, we would consider this evidence less relevant than one that records the actual use of the skills.

We judged the relevance of findings as either Very similar, Similar, Less similar, or Distant analogue.

Component 2: Relevance of context

We then assessed how relevant the context of the evidence piece is for the assumption that we were testing. 

To illustrate, consider the assumption Conservation practitioners use GIS skills to prioritise locations for new protected areas. If we found evidence related to training a particular target group in applying these specific skills, we would judge the relevance higher than evidence associated with other capacity-building efforts.

We considered the relevance of the context or action as either Very similar, Similar, Less similar, or Distant analogue.

Combining relevance of findings and of context

To judge the final relevance of an evidence piece, we combine the relevance of its findings with the relevance of its context using the following table: 

We judged evidence to be irrelevant if the relevance of the finding and the context were both ‘Distant analogue’. We excluded irrelevant evidence from further assessments.

Reliability

We defined two components that together determine the reliability of a piece of evidence: 1. rigour and appropriateness and 2. sample size. Both these components are related to the reliability of the information contained within the evidence. We did not consider the reliability of the source20.

Component 1: Rigour and appropriateness

We assessed the rigour by which information in the evidence piece was gathered and analysed. 

We used a range of questions, such as

  • Was the choice of methods appropriate for gathering the information? 
  • Were potential biases acknowledged and accounted for where possible? 
  • Were there appropriate counterfactuals or controls? 
  • Were multiple, complementary methods or sources used to gather the information?

We judged rigour and appropriateness as either High, Medium, Low or Nil.

Component 2: Sample size

We then assessed the volume of each evidence piece by looking carefully at its sample size. We viewed sample size as an essential component of reliability because we were testing general assumptions, i.e. not constrained by a particular context such as geography or type of organisation. Because of this broad applicability, a large sample size increases reliability.

To illustrate, let us consider a report providing a synthesis of all 100 known scientific publications on a particular topic. This evidence piece (the synthesis report) contains a high sample size (all 100 publications). Compare this with an evaluation report of one training. This evidence piece (the report) contains a low sample size (1 out of many).

We did not use thresholds for sample size because we found that the spectrum was enormous and varied between assumptions. Conservation strategies that are widely applied and have been applied for many years require a larger sample size than a conservation strategy that is relatively new and rarely applied. 

We judged the sample size as either High, Medium, Low or One.

Reliability based on both components

Once we had assessments of rigour and appropriateness, and sample size, we used the following table to determine the reliability of each piece of evidence:

Step 5 – Compile and conclude

Start by concluding to what extent the constituent assumptions of the learning question hold.

Once clear, interpret the results and formulate conclusions for the learning question. During this process, you will probably spot evidence gaps. Some of these might be important enough to consider in future learning initiatives.

Our approach to compiling the evidence and drawing conclusions

Compiling the evidence base

To understand the overall evidence for a particular assumption, we combined the individual pieces of evidence and displayed them in a summary Ziggurat plot.

A Ziggurat (or skyscraper) plot displays evidence pieces as horizontal blocks, organised in categories of the degree of support for the assumption (in line with the visualisation of degree of support and the weight as described in Step 4). The purpose of the plot is to show the balance of available evidence that supports or refutes the assumption.

The width of each block of evidence represents its weight. The maximum potential weight of a single block is one (very high weight), the minimum is 0.25 (low weight). The total weight of each category is calculated by adding up the weight of all its blocks.

Example Ziggurat plot for the assumption Being in a partnership has added value for the partners. Each piece of evidence is a horizontal block whose width represents its weight. The maximum potential weight of a single block is one. The number below each evidence block pile shows that pile’s total weight. To derive an average degree of support, we defined support categories as consecutive integers (refutes=1, mixed=2, weak=3, strong=4) and calculated the weighted mean (filled black point). You can imagine the filled black point as the location on a balance beam where you would need to place the fulcrum to balance both sides. The weighted mean was used as a guide for interpreting the overall degree of support, and not as a definitive answer. This is because where there is very variable or strongly bimodal support, the weighted mean may be a poor representation of the expected support for an assumption.

Drawing conclusions

To conclude on each assumption, we inspected the Ziggurat plots to see whether the balance of the evidence supported, refuted or showed mixed support for the assumption. A detailed consideration of all evidence pieces also allowed us to highlight important themes and discussion points.

Example from the learning topic Flexible funding. The combined evidence is showing overall strong support for the assumption (B1: Organisations use flexible funding to invest in organisational development and maturity).

In the example above, 34 out of 44 evidence pieces strongly support the assumption that organisations use flexible funding to invest in organisational development and maturity. Almost all combined weight of the evidence suggests either strong or some support for the assumption. So, in summary, the overall support for the assumption is strong.

When the balance of evidence did not support or refute the assumption, we considered whether the two different sources of evidence (i.e. evidence from inside and outside the MAVA realm) presented different conclusions. In some cases, support for the assumption differed between evidence from MAVA’s grant portfolio and evidence from the broader literature. By highlighting key themes, we hoped to explain the reasons for these differences. 

Example from the learning topic Research and monitoring. The combined evidence is showing overall mixed support for the assumption (A1: Conservation practice is aligned with and informed by research findings). However, the evidence from MAVA (darker blocks) shows overwhelming support, while the evidence from other sources (lighter blocks) tends to refute or be mixed.

In this other example above, the combined evidence shows mixed support for the assumption that conservation practice is aligned and informed by research findings. In this case, the evidence both supports and refutes the assumption, though the weight of the supporting side is stronger. However, there is a significant difference between evidence from the MAVA domain and the broader literature. All MAVA-related evidence shows some or strong support for the assumption. The vast majority of other evidence refutes the assumption or suggests mixed support. In summary, the overall support for the assumption is mixed.

Finally, we combined the conclusions and learnings for each assumption to answer the related learning question. Where possible, we described existing evidence gaps to consider in further learning efforts. 

What we learned

After all this, what did we find about our learning topics? See the overview or investigate the details on the learning topic pages: Capacity-building, Partnerships and alliances, Flexible funding, and Research and monitoring. 

Join the learning

This first attempt at a 5-step approach to evidence-based learning has been developed and tested on four learning topics. Foundations of Success and Conservation Evidence intend to revise and improve it over time – ideally with your participation! If you want to share your thoughts and engage with us about the approach and your learning topics, please contact us.

We are planning exciting things to continue learning from evidence, including online events and new learning topics. Sign up here or email us at info@conservation-learning.org to receive updates.

Links

Learning topics

Download report

Conservation Learning Initiative (PDF, 18 MB)

Web links

References

  1. Christie, A. P., Abecasis, D., Adjeroud, M., Alonso, J. C., Amano, T., Anton, A., Baldigo, B. P., Barrientos, R., Bicknell, J. E., Buhl, D. A., Cebrian, J., Ceia, R. S., Cibils-Martina, L., Clarke, S., Claudet, J., Craig, M. D., Davoult, D., De Backer, A., Donovan, M. K., … Sutherland, W. J. (2020). Quantifying and addressing the prevalence and bias of study designs in the environmental and social sciences. Nature Communications, 11(1), 6377. https://doi.org/10.1038/s41467-020-20142-y
  2. Christie, A. P., Chiaravalloti, R. M., Irvine, R., Kincaid, K., Morgan, W., Rezaie, A. M., Salafsky, N., Sutherland, W. J., White, T. B., & Yamashita, H. (2022). Combining and assessing global and local evidence to improve decision-making. OSF. [In prep].
  3. Conservation Measures Partnership. (2004). Open Standards for the Practice of Conservation Version 1.0. CMP.
  4. Conservation Measures Partnership. (2007). Open Standards for the Practice of Conservation Version 2.0. CMP.
  5. Conservation Measures Partnership. (2013). Open Standards for the Practice of Conservation Version 3.0. CMP.
  6. Conservation Measures Partnership. (2016). Conservation Actions Classification v 2.0. https://docs.google.com/spreadsheets/d/1i25GTaEA80HwMvsTiYkdOoXRPWiVPZ5l6KioWx9g2zM/edit#gid=874211847
  7. Conservation Measures Partnership. (2020). Open Standards for the Practice of Conservation Version 4.0. CMP.
  8. Dubois, N., & Gómez, A. (2018). Evidence in Action. Using and generating evidence about effectiveness in biodiversity programming. United States Agency for International Development (USAID).
  9. Margoluis, R., Stem, C., Swaminathan, V., Brown, M., Johnson, A., Placci, G., Salafsky, N., & Tilders, I. (2013). Results Chains: a Tool for Conservation Action Design, Management, and Evaluation. Ecology and Society, 18(3). https://doi.org/10.5751/ES-05610-180322
  10. Moon, K., Blackman, D. A., Adams, V. M., Colvin, R. M., Davila, F., Evans, M. C., Januchowski-Hartley, S. R., Bennett, N. J., Dickinson, H., Sandbrook, C., Sherren, K., St. John, F. A. V., van Kerkhoff, L., & Wyborn, C. (2019). Expanding the role of social science in conservation through an engagement with philosophy, methodology, and methods. Methods in Ecology and Evolution, 10(3), 294–302. https://doi.org/10.1111/2041-210X.13126
  11. Pullin, A. S., & Knight, T. M. (2001). Effectiveness in Conservation Practice: Pointers from Medicine and Public Health. Conservation Biology, 15(1), 50–54. https://doi.org/10.1111/j.1523-1739.2001.99499.x
  12. Pullin, A. S., Knight, T. M., Stone, D. A., & Charman, K. (2004). Do conservation managers use scientific evidence to support their decision-making? Biological Conservation, 119(2), 245–252. https://doi.org/10.1016/j.biocon.2003.11.007
  13. Salafsky, N., Margoluis, R., Redford, K. H., & Robinson, J. G. (2002). Improving the Practice of Conservation: a Conceptual Framework and Research Agenda for Conservation Science. Conservation Biology, 16(6), 1469–1479. https://doi.org/10.1046/j.1523-1739.2002.01232.x
  14. Salafsky, N., Salzer, D., Stattersfield, A. J., Hilton-Taylor, C., Neugarten, R., Butchart, S. H. M., Collen, B., Cox, N., Master, L. L., O’Connor, S., & Wilkie, D. (2008). A Standard Lexicon for Biodiversity Conservation: Unified Classifications of Threats and Actions. Conservation Biology, 22(4), 897–911. https://doi.org/10.1111/j.1523-1739.2008.00937.x
  15. Salafsky, N., Boshoven, J., Burivalova, Z., Dubois, N. S., Gomez, A., Johnson, A., Lee, A., Margoluis, R., Morrison, J., Muir, M., Pratt, S. C., Pullin, A. S., Salzer, D., Stewart, A., Sutherland, W. J., & Wordley, C. F. R. (2019). Defining and using evidence in conservation practice. Conservation Science and Practice, 1(5), e27. https://doi.org/10.1111/csp2.27
  16. Salafsky, N., & Margoluis, R. (2021). Pathways to success: taking conservation to scale in complex systems. Island Press. ISBN 9781642831368
  17. Salafsky, N., Irvine, R., Boshoven, J., Lucas, J., Prior, K., Bisaillon, J., Graham, B., Harper, P., Laurin, A. Y., Lavers, A., Neufeld, L., & Margoluis, R. (2022). A practical approach to assessing existing evidence for specific conservation strategies. Conservation Science and Practice, 4(4). https://doi.org/10.1111/csp2.12654
  18. Suter, G. W. (2016). Weight of Evidence in Ecological Assessment. United States Environmental Protection Agency.
  19. Sutherland, W. J., Pullin, A. S., Dolman, P. M., & Knight, T. M. (2004). The need for evidence-based conservation. Trends in Ecology & Evolution, 19(6), 305–308. https://doi.org/10.1016/j.tree.2004.03.018
  20. Sutherland, W. J. (2022). Transforming Conservation: A practical guide to evidence and decision making [In prep].