Finding what works

H ow do we achieve coordinated, evidence-based action to meet the goal of Zero Hunger? How do we provide policy and decision makers with evidence about innovative solutions from research that exists now? How can we infuse policy with science so decision makers can quickly allocate resources to address urgent problems—and in a way that does not cherry-pick results or minimize the complexity of science?

We ask these questions against a complicated backdrop: There is more information available than we can possibly manage; science is increasingly specialized; and access to scientific research has become so restricted it has made the use of science problematic. These barriers, high to scientists, are even higher for non-research audiences.

STEP 1

In order to deliver evidence-based information, we first have to find it, connect it, and classify it. The research we need to look at is scattered across disparate silos with no connection to each other. We’re using machine learning and natural language processing to unify this knowledge in such a way that we can explore large volumes of text quickly to create the connections between research and policy.

EXAMPLE 1: SYNONYMS FOR INTERVENTION

In agricultural research, the word intervention isn’t used consistently as a way of describing “what works” to address a particular problem, even though there are, in reality, many interventions designed to tackle agricultural and food security problems that have been researched.

Semantic associations, or synonyms, are a key building block for us. They enrich our ability to discover what we are looking for and what we hope to find. For example, when we conduct a keyword search using only “interventions” and “greenhouse gas emissions” in our dataset, we find only about 10 percent of the relevant materials. But when we create associations between interventions and possible synonyms, this increases to 61 percent of the dataset.

Now that we knew how interventions were described in agricultural research, we could set about analyzing our sample of articles to find and classify specific interventions. 

We found synonyms by looking at hypernyms and hyponyms, which are a type of relationship in semantics; for instance, a lemon is a hyponym of fruit, and classified them into four broad categories—technical, socioeconomic, ecosystem, unclassified—and then, specifically, as 995 narrow intervention concepts.

This is a much more targeted approach to uncover important research. It gives us a wholly new way to classify and organize sciences that it is accessible to an audience interested in policy-relevant research. See our Ceres2030 dashboard page for more details.

EXAMPLE 2: TOPIC MODELING AND GAPS IN EVIDENCE

Natural language processing also enabled us to unify and explore data, despite the fact the data came from many different places. We used topic modeling—a way of exploring text to see what is has in common with other text in the same corpus—to establish a baseline from which we could map the evolution of research from 2008 to 2018.

We can see the topics where there was a high level of research (the darker the blue in the image below, the greater the density of research papers) or where evidence and research were limited or missing. We can also create comprehensive research baselines to see the volume of research by topic, by funder, by country, and the potential relevancy of the research.

STEP 3Evidence reviews, like scoping and systematic reviews, bring all the studies on a particular issue or intervention together to evaluate what they mean. It’s a process with specific steps designed to minimize bias and to ensure rigor and transparency, so that someone else could replicate the process and reach the same conclusion. In this process, each of the eight intervention review teams will be supported by research synthesis experts to help guide them through the process.

The first critical step in this will be for the authors to create a protocol for each review. This is the roadmap setting out how the review is going to be done, how the reviewers will decide what studies or data to include or exclude in the review, and how those studies and data will be reviewed.

This is the opportunity to have open, critical, and constructive discussion among the invited reviewers, synthesis research experts, and global stakeholders, before any actual review of the evidence takes place. Our goal is maximum engagement and maximum integrity.

One particular issue facing agricultural research is that it has fewer randomized control trials than, say, medicine, and needs to be inclusive of many different kinds of evidence and data. This makes scientific appraisal more difficult.

For this reason, we are taking a mixed-methods approach, which combines quantitative and qualitative evidence on complex and pressing questions, and which has been successful in previous agricultural systematic reviews.