What is evidence synthesis?

When it comes to urgent and complex problems such as ending hunger, we need solutions as quickly as possible, and we need to know that they work. The good news is that there has never been more research on these problems—and each day, that stock of knowledge expands as new studies and reports are published.

The problem is synthesizing what all this knowledge means. It can take a year to track down the relevant research on a particular question or problem—because there is so much of it scattered across so many repositories and domains.

Evidence synthesis is an umbrella term describing the process of creating a summary of the underlying information. It uses a carefully worked-out methodology to evaluate interventions. Evidence synthesis is not new and there are many different types of evidence synthesis: systematic reviews are probably the best known. The need for evidence synthesis is acute,  the more research we produce, the more we need to assess and incorporate the findings of new studies.

[How our research teams are assessing the quality of the evidence—an explainer] 

In agriculture, the past decade has generated more than two-million articles across hundreds of academic journals. More than 60 major agencies publish reports and studies. There is no keyword search for “what works.” There are no meta-tags for specific policy interventions.

This is why we turned to machine learning. We created a searchable evidence map from all this research, from which we could identify and classify the kinds of agricultural interventions that addressed the key elements of SDG 2. In a sense, machine learning gave a voice to this research so it could tell us whether it could be relevant for policy makers. 

[How we used machine learning to create an evidence map for agricultural research]

We could see the most studied interventions using the evidence map. We asked our global advisory board of experts to explore the data with us, and to help pick the most important interventions—those that should go on for detailed academic review to better understand how they work, and in what contexts.

One of the challenges of reviewing evidence in agriculture is that almost everything works in some context. Unlike biomedicine, where evidence for “what works” can be examined in carefully controlled clinical isolation, evidence in agriculture is influenced by many diverse economic, geographical, and social factors. Our research questions had to be broad enough to account for these factors yet precise enough to reveal the kind of robust evidence for ‘what works’ that people can use to make investment and policy decisions.

The conclusions of these evidence syntheses will, subject to peer review, be published as a special collection in Nature Research Journals in 2020.  

How we assess the quality of evidence

Evidence syntheses, like scoping and systematic reviews, bring all the studies on a particular issue or intervention together to evaluate what they mean. It’s a process with specific steps designed to minimize bias and to ensure rigor and transparency, so that someone else could replicate the process and reach the same conclusion.

Each of the eight intervention research teams is supported by research synthesis experts. The first critical step is for the authors to create a protocol for each review. This is the roadmap setting out how the review is going to be done, how the reviewers will decide what studies or data to include or exclude in the review, and how those studies and data will be reviewed.

One particular issue facing agricultural research is that it has fewer randomized control trials than, say, medicine, and needs to be inclusive of many different kinds of evidence and data. This makes scientific appraisal more difficult. For this reason, we are taking a mixed-methods approach, which combines quantitative and qualitative evidence on complex and pressing questions, and which has been successful in previous agricultural systematic reviews. Groups with deep experience in mixed-methods reviews, such as the Campbell Collaboration and the Center for Evidence-Based Agriculture have worked closely with us to explore appropriate methods and offer expert advice.

Once each research team reached a consensus on the protocol, it was published—and it cannot be changed. Publishing ahead of doing the review protects its replicability and transparency. It also gives us a chance to share our work, what we are doing, and allow for scientific dissent as part of the process.

All the protocols have been uploaded to the OSF open-science platform. You can find links to these on each question pag

Using machine learning to find
policy interventions to end hunger

Given the amount of information available, relying only on keyword searching doesn’t work. If we want to understand the fullness of human knowledge, we need to incorporate new methods of discovery that account for the way we describe similar things in different ways.

Over the past decade, there have been enormous advances in artificial intelligence that enable computers to analyze the way we use language. This involves training a computer program to recognize relationships between words, so that it can capture the different ways people describe similar things.

We used machine learning and natural language processing (NLP) to create and analyze a preliminary dataset of ~50,000 articles and reports (2008-2018) about smallholder farmers from science journals and research and development organizations. We used a variety of search terms, such as small-scale food producers, rural farmers, and subsistence and contract farmers.

In order to increase coverage of materials published in low and middle income countries, we included the full table of contents from the African Journal of Biotechnology, African Journal of Agricultural Research, African Journal of Food, Agriculture, Nutrition and Development, African Crop Science Journal, Indian Journal of Agronomy, and the Indian Journal of Agricultural Economics.