Can Kolibri Fly in a Refugee Settlement? Integrating Voices in a Project-Based Learning (PBL), Foundational Literacy and Numeracy (FLN) tech-enabled evaluation with out of school learners in Uganda

Abstract Information: Project-based learning (PBL) has demonstrated some effectiveness in improving foundational skills but is typically limited to resource-rich environments given the tools and knowledge needed for its implementation. Three organizations co-designed a program to support out-of-school refugee learners in Uganda in using technology-enabled education without the Internet as an enabler for this method of learning. This evaluation shows how Internet-free education technology in low-resource learning environments, when designed and implemented in a thoughtful and relevant way, can amplify playful and PBL to accelerate foundational literacy, numeracy (FLN), and social-emotional learning (SEL) skills.

First, the evaluation design, driven by a strong theory of change, demonstrated a promising approach to measure the effectiveness of PBL for FLN in a blended learning context. These types of gains open new avenues of thinking for the potential for PBL approaches in low-resource contexts. Specifically, the evaluation had a pre-/post-test design over two cohorts with program design changes made between them. Qualitative data indicate that teachers and learners improved in English literacy, numeracy, social-emotional skills (including self-confidence, socializing and effective team-work), and practical life skills (including gardening). Quantitative learner assessment data, on measures adapted from Teaching at the Right Level, demonstrated a significant effect on FLN learning outcomes. By the end, all learners (n=714) improved in reading, with the ability to read a word went from 20% of learners to 100%, and 73% could read a paragraph by the end. The ability to identify a missing number went from 36% to 97%, with 72% doing 2- digit addition and subtraction. There was a statistically significant increase in scores at the 95% confidence level with large effect sizes. At endline, quantitative SEL scores also saw a statistically significant increase with large effect size. Furthermore, time spent engaging with the education technology was correlated with higher learning outcomes.

Second, the collection of data was unique for an edtech intervention for an undeserved population. It was presumed that data collected through the edtech would be very valid and reliable, but the model of the intervention with a 2:1 learner to tablet ratio and different login accounts made it difficult to track individual-level usage of content. Despite training, teachers did not fill out implementation logs consistently, nor was there high interrater reliability for classroom observations, resulting in large gaps in process evaluation data. However, data to inform the outcome evaluation was excellent - primarily through quantitative baseline and endline learning assessments, surveys, and focus groups.

Third, this approach integrated expert evidence partner voices with novice evaluators in the field, and presented opportunities and interesting challenges. The expert evidence partner was critical of data that were not focused on the outdated concept of “fidelity of implementation.” On the other hand, the implementing organization’s skills were generally less robust in collecting implementation-related data, but very effective in collecting outcome data and providing important feedback when co-designing data collection tools. This collaboration increased local capacity for data collection, but also reiterated the importance of integrating evaluative expertise with an understanding of the contextual realities.