Predicting High Impact Grants for Case Study Selection

Abstract Information: This paper explores a methodology to identify grants with high levels of social and economic impact using mathematical modelling instead of relying on expert peer review.

Understanding Impact

Understanding the impact of research has long been a challenge for funders wishing to understand and improve their ability to generate real-world impact.

1. Bibliometrics
Most organisations use readily available bibliometric measures as proxies for impact but recognise that high citation counts are not a good indicator for the social and economic impact of research.

2. Case Studies
Some organisations conduct more in-depth, qualitative evaluations, but there is a challenge in how to scale this approach across a large research portfolio, or across organizations and time periods. One example of this is the UK Research Excellence Framework (REF), which is a national exercise conducted at several year intervals to allocate significant amounts of government funding to universities. The precise rules have varied each time the REF has been run, but one of the main components is the requirement to submit a number of Impact Case Studies (ICS). These case studies are assessed by peer review and scored, with higher scores leading to increased funding for the university. As the results of this exercise are very consequential, a considerable amount of time and money is spent to identify and evidence impact produced through research at the university over an extended period.

3. Mixed Methods
Other organizations, particularly in the UK, use Researchfish (RF) as a repeated measure survey tool. RF has operationalisations of a range of constructs specifically designed to facilitate an understanding of social and economic impact using quantitative and qualitative methods. This allows funders to understand the outputs, outcomes and impacts of their research more easily across an extended portfolio.

Study

This study uses publicly available data on the Impact Case Studies submitted as part of the most recent REF and applies a number of data enrichment techniques to develop connections between those ICS and grants that have reported using RF. This extends the work previously published in https://doi.org/10.12688/f1000research.74374.3 Once the data has been enriched a probit model is used to predict the likelihood of a particular grant being used as part of an ICS, test a logic model for impact, and develop a number of variables to describe characteristics of the grant and its outputs.

Implications

The ability to use relatively light touch existing data to simulate the results of a more costly and labour-intensive peer review process in identifying high impact areas of a portfolio is useful for organisations seeking to systematically identify areas of large portfolios engaged in producing high levels of social and economic impact. The significant variables identified by the model might also help to confirm or challenge a priori assumptions about the sorts of indicators that might be useful in identifying high impact grants. The approach can also support impact case study creation by “zooming in” on areas of interest, winnowing the available data and allowing expert time to be better spent.