Approach to evidence-based conservation learning
Explore the five practical steps for evidence-based learning in conservation.
In this initiative, we use evidence to test critical assumptions in conservation practice. We have elaborated five practical steps for using assumptions and evidence to answer essential learning questions.
The approach we present here is a first attempt at formulating a method for evidence-based learning. We are continuously improving our process based on its application in practice and input from the conservation community.
By following this approach and optimising evidence use, it is possible to unveil crucial lessons about conservation strategies.
Five steps for evidence-based learning
The core elements of evidence-based learning are: a well-defined learning topic corresponding to a conservation action, relevant learning questions that help to learn about the mechanics of the action, and explicit assumptions to test with evidence. The five steps for evidence-based learning focus on and utilise these elements:
- Define the overall learning topic
- Develop learning questions and formulate assumptions for each
- Collect evidence related to the assumptions
- Assess the evidence to determine whether it supports the assumptions
- Compile and conclude what we have learned about the question
Already in the early 2000s, the conservation community identified the need to use evidence for decision-making to increase the effectiveness of conservation strategies11,12,19. Since then, the efforts to establish an evidence-based conservation practice have been twofold:
- Published evidence: Approaches have emerged to collect and compile existing published evidence18 and to make it accessible for conservation practitioners through online libraries and databases. Examples of these efforts are the CEEDER database of the Collaboration for Environmental Evidence (CEE) and Conservation Evidence at the University of Cambridge.
- Evidence from practice: The conservation sector became more rigorous in project and programme design and management. The aim was to improve conservation practice through adaptive management13 and standard practices3,4,5,7.
Building on these developments, the conservation community has recently developed concepts for defining and using evidence in practice8,15,16 and proposed an approach to assess practical evidence for specific conservation strategies17.
The five-step approach we are presenting here heavily builds on these developments over the last 20 years.
Step 1 – Define the learning topic
Define the topic for learning, why learning about this topic is essential, and who is (likely) interested in the lessons learned. This step forms the context for the other steps of the process.
Our approach to defining the learning topic
Our first challenge was to select a handful of learning topics from the MAVA portfolio of conservation grants. The portfolio spans roughly 1700 grants, addressing numerous conservation issues across different geographic regions and nearly 30 years.
Potential learning topics: We started with a quick scan of all 1700 grants. For each grant, we identified what main conservation strategies it covered. We then cross-referenced these strategies using a standard classification scheme of conservation actions14,6. This helped us list roughly 100 different conservation strategies implemented through the MAVA portfolio.
Potential evidence: We then narrowed the list down to 14 conservation strategies for which we expected evidence to be available. For this, we looked at the total number of grants that included the strategy and the total financial investment by MAVA in those grants. We assumed that the higher the total number of grants and the higher the total financial investment – the more likely it would be that we would find enough evidence.
Relevance and interest: Finally, we ranked each of the remaining 14 strategies by evaluating how relevant the learning topic is to the broader conservation community. We combined this ranking with our interest in each of them as learning topics.
Through this process, we arrived at four selected learning topics:
- Building capacity of key stakeholders and conservation organisations
- Establishing strategic conservation partnerships and alliances
- Providing flexible conservation funding
- Basic conservation research and monitoring
Step 2 – Develop learning questions
Find focus in your learning topics by formulating relevant learning questions and associated assumptions that you can test with evidence. You can build the necessary framework for selecting learning questions by creating a theory of change that captures the pathway from conservation action to desired outcomes and impact. This step helps you to focus on desired learning outcomes.
Our approach to developing learning questions
For each of the four learning topics, we developed a theory of change . We used a results chain diagram9 to explain how we assumed the conservation action would lead to specific results. Whenever possible, we started with a generic results chain from the Conservation Actions & Measures Library (CAML). We then refined this generic chain through various rounds of reviewing and validation to finally arrive at a version that best represents the work implemented through MAVA-funded grants.
We used the results and the assumed causal relationships between results to tease out essential learning questions . We then formulated one or more assumptions for each learning question to test with evidence.
Step 3 – Collect Evidence
Our approach to collecting evidence
We collected all the relevant evidence for each assumption in the MAVA domain. We complemented this with evidence collected from outside the MAVA domain.
Looking beyond the MAVA domain helped us widen the evidence base and look for generic lessons learned. The amount of evidence we collected from outside the MAVA domain was limited by the available time, i.e. we did not do exhaustive reviews of all published evidence.
We provide a detailed description of the evidence base for all learning topics in the Learning section.
Sources inside the MAVA domain:
- Questionnaires to MAVA grantees
- Discussions and focus groups with the MAVA team
- MAVA grantee reports, proposals, and evaluations
Sources outside the MAVA domain:
- Conservation Evidence database through systematic searches
- Google and Google Scholar through exploratory searches
- Discussions with key researchers to highlight additional sources
The different types of evidence we found:
- Responses to questionnaires – both qualitative and quantitative
- Processed financial data
- Quotes and extracts from reports and articles
- Results and findings from research articles
- Conclusions from discussions with expert groups
Step 4 – Assess Evidence
It is essential to maintain consistency in your assessment of reliability and relevance. Group discussions, consultation with experts, and multiple iterations help ensure consistency.
Our approach to assessing evidence
To test our assumptions, we looked at every piece of evidence individually. We assessed the degree to which it supports the assumption and the weight of the evidence. To determine the weight, we looked at the reliability and relevance of the evidence17,2. In formulating this part of the approach, we relied heavily on the work of Salafsky et al., 201915.
We had repeated discussions to maintain consistency in our assessments across evidence pieces and learning topics. By working through numerous examples, we established what features are typical for the most vs least reliable evidence pieces and the most vs least relevant ones. Where helpful, we wrote down general descriptors for each level of reliability and relevance to serve as reference material when assessing evidence.
Degree of support
We took a two-step approach to determine the degree of support.
First, we considered whether a piece of evidence indicates that the assumption is true (i.e. supports it), false (i.e. refutes it), or a mix of both (i.e. shows mixed support).
Second, if the piece of evidence supports the assumption, how strongly does it do so: does it show some support or strong support? On the negative side, in contrast, it was difficult to distinguish between strength of refutation in this way.
We therefore decided to judge the support of an individual piece of evidence to a particular assumption as either Strong, Some, Mixed, or Refuting. The degree of support can be illustrated by the colour and symbol of the evidence piece.
To illustrate this, consider the following assumption:
Trained people perform better than untrained people
A study that found improved performance following training would support this assumption. Furthermore, if the vast majority of trainees showed improved performance, support for the assumption would be strong. On the other hand, for a different study showing that only around half of the trainees improved, the support would be mixed. The assumption would be refuted by a study that found either that trained and untrained people perform equally well or that untrained people perform better.
The exact formulation of the assumption greatly influences the degree of support. For example, an alternative assumption to the one above could be There is no difference in performance between trained and untrained people. In this case, the assumption would be refuted by evidence for any difference between the two groups, regardless if it showed trained or untrained people performed better. However, a study showing that they perform equally well would support it.
To assess the weight of a piece of evidence, we considered its relevance and reliability.
Relevance refers to whether the evidence piece should be taken into account at all. Evidence pieces with low relevance to the assumption might be considered, but will have lower weight. Similarly, evidence pieces with low reliability also reduce the weight of evidence, regardless of its conclusion.
We used four weight categories: Very high, High, Medium and Low. The weight can be illustrated by the width of the evidence piece: the wider, the heavier.
The table below shows how we combine relevance and reliability into weight.
We identified two components that together determine the relevance of a piece of evidence: relevance of findings and relevance of context.
Component 1: Relevance of findings
We first assessed how relevant the findings of the evidence piece were to the assumption that we were testing. We judged the relevance of the findings high if we could meaningfully connect the findings to the assumption. We considered the relevance lower if we needed to make an additional assumption.
To illustrate, consider the assumption Trainees apply their skills in their ongoing work. Imagine that a piece of evidence found that a group of trainees feels highly motivated to use their new skills following a training course, but without recording whether they also use the skills. We could use this evidence to assess the assumption, but if we do, we must also assume that this motivation translates into trainees using the skills. Therefore, we would consider this evidence less relevant than one that records the actual use of the skills.
We judged the relevance of findings as either Very similar, Similar, Less similar, or Distant analogue.
Component 2: Relevance of context
We then assessed how relevant the context of the evidence piece is for the assumption that we were testing.
To illustrate, consider the assumption Conservation practitioners use GIS skills to prioritise locations for new protected areas. If we found evidence related to training a particular target group in applying these specific skills, we would judge the relevance higher than evidence associated with other capacity-building efforts.
We considered the relevance of the context or action as either Very similar, Similar, Less similar, or Distant analogue.
Combining relevance of findings and of context
To judge the final relevance of an evidence piece, we combine the relevance of its findings with the relevance of its context using the following table:
We judged evidence to be irrelevant if the relevance of the finding and the context were both ‘Distant analogue’. We excluded irrelevant evidence from further assessments.
We defined two components that together determine the reliability of a piece of evidence: 1. rigour and appropriateness and 2. sample size. Both these components are related to the reliability of the information contained within the evidence. We did not consider the reliability of the source20.
Component 1: Rigour and appropriateness
We assessed the rigour by which information in the evidence piece was gathered and analysed.
We used a range of questions, such as
- Was the choice of methods appropriate for gathering the information?
- Were potential biases acknowledged and accounted for where possible?
- Were there appropriate counterfactuals or controls?
- Were multiple, complementary methods or sources used to gather the information?
We judged rigour and appropriateness as either High, Medium, Low or Nil.
Component 2: Sample size
We then assessed the volume of each evidence piece by looking carefully at its sample size. We viewed sample size as an essential component of reliability because we were testing general assumptions, i.e. not constrained by a particular context such as geography or type of organisation. Because of this broad applicability, a large sample size increases reliability.
To illustrate, let us consider a report providing a synthesis of all 100 known scientific publications on a particular topic. This evidence piece (the synthesis report) contains a high sample size (all 100 publications). Compare this with an evaluation report of one training. This evidence piece (the report) contains a low sample size (1 out of many).
We did not use thresholds for sample size because we found that the spectrum was enormous and varied between assumptions. Conservation strategies that are widely applied and have been applied for many years require a larger sample size than a conservation strategy that is relatively new and rarely applied.
We judged the sample size as either High, Medium, Low or One.
Reliability based on both components
Once we had assessments of rigour and appropriateness, and sample size, we used the following table to determine the reliability of each piece of evidence:
Step 5 – Compile and conclude
Start by concluding to what extent the constituent assumptions of the learning question hold.
Once clear, interpret the results and formulate conclusions for the learning question. During this process, you will probably spot evidence gaps. Some of these might be important enough to consider in future learning initiatives.
Our approach to compiling the evidence and drawing conclusions
Compiling the evidence base
To understand the overall evidence for a particular assumption, we combined the individual pieces of evidence and displayed them in a summary Ziggurat plot.
A Ziggurat (or skyscraper) plot displays evidence pieces as horizontal blocks, organised in categories of the degree of support for the assumption (in line with the visualisation of degree of support and the weight as described in Step 4). The purpose of the plot is to show the balance of available evidence that supports or refutes the assumption.
The width of each block of evidence represents its weight. The maximum potential weight of a single block is one (very high weight), the minimum is 0.25 (low weight). The total weight of each category is calculated by adding up the weight of all its blocks.
To conclude on each assumption, we inspected the Ziggurat plots to see whether the balance of the evidence supported, refuted or showed mixed support for the assumption. A detailed consideration of all evidence pieces also allowed us to highlight important themes and discussion points.
In the example above, 34 out of 44 evidence pieces strongly support the assumption that organisations use flexible funding to invest in organisational development and maturity. Almost all combined weight of the evidence suggests either strong or some support for the assumption. So, in summary, the overall support for the assumption is strong.
When the balance of evidence did not support or refute the assumption, we considered whether the two different sources of evidence (i.e. evidence from inside and outside the MAVA realm) presented different conclusions. In some cases, support for the assumption differed between evidence from MAVA’s grant portfolio and evidence from the broader literature. By highlighting key themes, we hoped to explain the reasons for these differences.
In this other example above, the combined evidence shows mixed support for the assumption that conservation practice is aligned and informed by research findings. In this case, the evidence both supports and refutes the assumption, though the weight of the supporting side is stronger. However, there is a significant difference between evidence from the MAVA domain and the broader literature. All MAVA-related evidence shows some or strong support for the assumption. The vast majority of other evidence refutes the assumption or suggests mixed support. In summary, the overall support for the assumption is mixed.
Finally, we combined the conclusions and learnings for each assumption to answer the related learning question. Where possible, we described existing evidence gaps to consider in further learning efforts.
What we learned
After all this, what did we find about our learning topics? See the overview or investigate the details on the learning topic pages: Capacity-building, Partnerships and alliances, Flexible funding, and Research and monitoring.
Join the learning
This first attempt at a 5-step approach to evidence-based learning has been developed and tested on four learning topics. Foundations of Success and Conservation Evidence intend to revise and improve it over time – ideally with your participation! If you want to share your thoughts and engage with us about the approach and your learning topics, please contact us.
- Overview of all learning topics
- Partnerships and alliances
- Flexible funding
- Research and monitoring
Conservation Learning Initiative (PDF, 18 MB)
- Conservation Evidence Database – Summaries of evidence from the scientific literature about the effects of conservation actions
- Conservation Actions and Measures Library (CAML) – An open-source library housing generic theories of change for conservation actions
- Collaboration for Environmental Evidence (CEE) – Seeks to promote and deliver evidence syntheses on issues of greatest concern to environmental policy and practice
- MAVA Learning – Practical insights of value to the community of funders as well as to the larger conservation community
- Christie, A. P., Abecasis, D., Adjeroud, M., Alonso, J. C., Amano, T., Anton, A., Baldigo, B. P., Barrientos, R., Bicknell, J. E., Buhl, D. A., Cebrian, J., Ceia, R. S., Cibils-Martina, L., Clarke, S., Claudet, J., Craig, M. D., Davoult, D., De Backer, A., Donovan, M. K., … Sutherland, W. J. (2020). Quantifying and addressing the prevalence and bias of study designs in the environmental and social sciences. Nature Communications, 11(1), 6377. https://doi.org/10.1038/s41467-020-20142-y
- Christie, A.P., Morgan, W.H., Salafsky, N., White, T.B., Irvine, R., Boenisch, N., Chiaravalloti, R.F., Kincaid, K., Rezaie, A.M., Yamashita, H., Sutherland, W.J. (2023). Assessing diverse evidence to improve conservation decision-making. OSF.
- Conservation Measures Partnership. (2004). Open Standards for the Practice of Conservation Version 1.0. CMP.
- Conservation Measures Partnership. (2007). Open Standards for the Practice of Conservation Version 2.0. CMP.
- Conservation Measures Partnership. (2013). Open Standards for the Practice of Conservation Version 3.0. CMP.
- Conservation Measures Partnership. (2016). Conservation Actions Classification v 2.0. https://docs.google.com/spreadsheets/d/1i25GTaEA80HwMvsTiYkdOoXRPWiVPZ5l6KioWx9g2zM/edit#gid=874211847
- Conservation Measures Partnership. (2020). Open Standards for the Practice of Conservation Version 4.0. CMP.
- Dubois, N., & Gómez, A. (2018). Evidence in Action. Using and generating evidence about effectiveness in biodiversity programming. United States Agency for International Development (USAID).
- Margoluis, R., Stem, C., Swaminathan, V., Brown, M., Johnson, A., Placci, G., Salafsky, N., & Tilders, I. (2013). Results Chains: a Tool for Conservation Action Design, Management, and Evaluation. Ecology and Society, 18(3). https://doi.org/10.5751/ES-05610-180322
- Moon, K., Blackman, D. A., Adams, V. M., Colvin, R. M., Davila, F., Evans, M. C., Januchowski-Hartley, S. R., Bennett, N. J., Dickinson, H., Sandbrook, C., Sherren, K., St. John, F. A. V., van Kerkhoff, L., & Wyborn, C. (2019). Expanding the role of social science in conservation through an engagement with philosophy, methodology, and methods. Methods in Ecology and Evolution, 10(3), 294–302. https://doi.org/10.1111/2041-210X.13126
- Pullin, A. S., & Knight, T. M. (2001). Effectiveness in Conservation Practice: Pointers from Medicine and Public Health. Conservation Biology, 15(1), 50–54. https://doi.org/10.1111/j.1523-1739.2001.99499.x
- Pullin, A. S., Knight, T. M., Stone, D. A., & Charman, K. (2004). Do conservation managers use scientific evidence to support their decision-making? Biological Conservation, 119(2), 245–252. https://doi.org/10.1016/j.biocon.2003.11.007
- Salafsky, N., Margoluis, R., Redford, K. H., & Robinson, J. G. (2002). Improving the Practice of Conservation: a Conceptual Framework and Research Agenda for Conservation Science. Conservation Biology, 16(6), 1469–1479. https://doi.org/10.1046/j.1523-1739.2002.01232.x
- Salafsky, N., Salzer, D., Stattersfield, A. J., Hilton-Taylor, C., Neugarten, R., Butchart, S. H. M., Collen, B., Cox, N., Master, L. L., O’Connor, S., & Wilkie, D. (2008). A Standard Lexicon for Biodiversity Conservation: Unified Classifications of Threats and Actions. Conservation Biology, 22(4), 897–911. https://doi.org/10.1111/j.1523-1739.2008.00937.x
- Salafsky, N., Boshoven, J., Burivalova, Z., Dubois, N. S., Gomez, A., Johnson, A., Lee, A., Margoluis, R., Morrison, J., Muir, M., Pratt, S. C., Pullin, A. S., Salzer, D., Stewart, A., Sutherland, W. J., & Wordley, C. F. R. (2019). Defining and using evidence in conservation practice. Conservation Science and Practice, 1(5), e27. https://doi.org/10.1111/csp2.27
- Salafsky, N., & Margoluis, R. (2021). Pathways to success: taking conservation to scale in complex systems. Island Press. ISBN 9781642831368
- Salafsky, N., Irvine, R., Boshoven, J., Lucas, J., Prior, K., Bisaillon, J., Graham, B., Harper, P., Laurin, A. Y., Lavers, A., Neufeld, L., & Margoluis, R. (2022). A practical approach to assessing existing evidence for specific conservation strategies. Conservation Science and Practice, 4(4). https://doi.org/10.1111/csp2.12654
- Suter, G. W. (2016). Weight of Evidence in Ecological Assessment. United States Environmental Protection Agency.
- Sutherland, W. J., Pullin, A. S., Dolman, P. M., & Knight, T. M. (2004). The need for evidence-based conservation. Trends in Ecology & Evolution, 19(6), 305–308. https://doi.org/10.1016/j.tree.2004.03.018
- Sutherland, W. J. (2022). Transforming Conservation: A practical guide to evidence and decision making [In prep].