Capacity building toward meaningful measures for evaluation

Abstract Information: The proliferation of open data resources in recent years has unlocked the possibility of generating novel measurements of outcomes that are aligned with the mission of scientific institutions and funders. However, many of these data resources are heterogeneous and difficult to link not only with one another, but with internal institutional data (1). This panel brings together experts from academia and industry to present their ongoing research focused on developing computational, analytical, and predictive methods to enable capacity building for evaluation. Thorough analysis of DoD-funded research outcomes requires the ability to automatically perform large-scale structured and unstructured data linking. The ARI developed a centralized knowledge graph of DoD funded research by retrieving, cleaning, parsing, and indexing DoD research data gathered from publicly available sources. We used advanced NLP methods and entity extraction using the raw linkages provided in external structured PubMed XML records and Crossref to link DoD grants to publications in PubMed and CrossRef. Similarly, we integrated external USPTO data to accurately represent DoD support in the patent space. These data were generated with human-centered use cases in mind, and feedback from DoD program staff about barriers and opportunities to integrate data into decision-making will be incorporated. Prof. Ni’s work focuses on integrating linked data in order to identify challenges in addressing gender gaps in a wide variety of domains. Her work has examined gender disparities in scientific publications and citations (2). She has also quantified gender disparities in innovation and patenting dynamics (3). Prof. Ni will discuss potential mechanisms underlying these trends, including gender disparities in parenting burdens that are unequally distributed (4). She is currently working on projects to build the infrastructure for the study of the scientific workforce based on large-scale, heterogeneous and longitudinal data, aiming to identify variables that impede and facilitate the cultivation of a competitive scientific workforce. Prof. Hutchins’ recent work on outcome measures to build capacity for evaluation have focused on data and metric development, as well as downstream analysis of these measures for evaluation purposes. He developed the public-domain citation databases for the National Institutes of Health and developed metrics aligned with measuring outcomes aligned with the funding agency missions (5, 6, 7). One ongoing challenge with the development of meaningful outcome measures is the lack of data either because of incomplete coverage in open-source datasets or because of time delays in the ability to measure these. He will discuss recent work focused on using predictive analytics that could be used as a proxy for these unavailable data, or to predict their occurrence with high accuracy before they occur. He has developed measures that can provide likelihood estimates of a basic research paper being used and cited by downstream clinical research (6) or stimulating downstream innovation and patenting. More recent work has focused on predicting the likelihood that a scientific citation represents substantive knowledge transfer (8). Future work is focused on predicting whether scientific topic clusters are likely to yield transformative breakthrough or future FDA-approved therapeutics.