Strengthening local voices through analysis of impact communication stories written by community facilitators around the world and expanding their voice using AI

Abstract Information: Development agencies have long struggled to make effective use of qualitative data, and a heavy reliance on surveys means that outcomes are too often pre-determined in a top-down manner, with little space for the voice of community members and local field staff to shape the sector’s understanding of impact and success. A research team based at the University of East London has piloted a methodology to produce a more robust analysis of community data using thematic analysis supported by generative AI. Each year, each World Vision Area Programme (multi-year community development project) submits a “story of transformation” as a part of their annual report. These stories are written by community-based staff to capture a real example of what they consider to be a successful story of change in the life of a child in their community. They adhere to journalistic principles of consent and accuracy when writing these stories. In 2023, we conducted a thematic analysis of 1315 such stories that had been written in the previous year, beginning with pre-set codes and expanding the coding system as new themes emerged. Stories could be coded to multiple themes. The analysis revealed that the largest number of stories (42%) described an inner transformation in the hearts of children (coded as ‘hope’, but encompassing the sub-codes of aspirations, confidence, happiness, value change, dignity and safety). The other most common types of transformation described were: economic and material well-being (31%), enabling learning environment (19%), and relationship with families (9%). The findings capture the types of development impacts that local staff found most meaningful, and suggest that field staff place a high level of value on children’s inner well-being. Further analysis demonstrated that internal changes in children as well as relational changes, such as with their families or peers, were described by staff as strongly linked to more material changes such as basic living conditions and educational access. In 2024, the research team has repeated the exercise, seeking to duplicate the analysis using ChatGPT 4.0 (Team License). The team has used AI technology to code 1101 stories from 1050 APs, finding similar themes emerging but nonetheless evidencing a slight shift in the type of impact field staff seek to have in their work and see as most worth celebrating.