'Bridging Gaps: Developing an Inclusive Monitoring, Evaluation, and Learning (MEL) Toolkit for Offline-First Education Technology'

Abstract Information: In under-resourced learning environments, where access to the internet is scarce, the effectiveness of education technology interventions is often compromised by challenges in data collection and utilization. Addressing this issue head-on, Learning Equality introduces Kolibri—a comprehensive ecosystem of free, open-source, adaptable tools designed for offline-first education. Kolibri enables the collection of a wide range of data, including user engagement metrics (time spent on the platform, interactions with specific content), academic performance (quiz scores, mastery-based learning progress), and educator tool usage. However, while Kolibri provides selective data visibility to learners, educators, and administrators to meet various needs, guidance on leveraging this data for program design and enhancement, and impact evaluation is lacking.

Globally, organizations implement Kolibri without Learning Equality’s direct support enabled by the Kolibri Edtech Toolkit, which includes instructional materials, training guides, and comprehensive documentation. Through this do-it-yourself approach, it has reached an estimated 3 million learners across 220+ countries and territories. This year marks a significant expansion of the Toolkit, with the introduction of the new Monitoring, Evaluation, and Learning (MEL) Toolkit. This enhancement aims to empower organizations to capture both qualitative and quantitative data within and beyond the Kolibri platform, facilitating informed decision-making and evaluation of Kolibri-based interventions. This MEL Toolkit will provide guidance for implementing organizations and other actors (such as government) to enhance the rigor of evaluation design and validity and reliability of the data collected. It will also provide suggestions for analysis and even sample data collection tools and approaches in offline contexts. Not only will this help data-driven decision making for ongoing programming at the local level, but has the potential to feed data back to Learning Equality, thereby allowing it greater opportunity to gather data and potentially do future cross-context analyses.

The MEL Toolkit is tailored for effective data collection in low-connectivity settings, emphasizing continuous refinement to bridge crucial data gaps. The ultimate goal is to enable organizations involved in educational interventions to take ownership of their own data and make data-driven decisions that enhance learning outcomes. The intention is that this Toolkit can be informed by the needs of new evaluators, and ensure that those from underserved groups have the skills they need to conduct MEL on behalf of their organization.

This roundtable will explore the challenges organizations face in conducting monitoring and evaluation with limited expertise in this area as well as resources. It will share the initial tools and materials developed for the Toolkit and ask key questions to elicit feedback that can provide its ongoing development. It draws from Learning Equality’s experiences to date in mentoring new evaluators and upskilling them on collecting data related to usage of Kolibri. Through this roundtable, we seek a dynamic discussion on the challenges and opportunities of developing a set of materials for a MEL Toolkit including how to shift the narrative away from expert evaluators coming in for a transactional study and towards rigorous, yet authentic, and contextually relevant evaluation centering diverse voices; and explore the transformative potential of inclusive evaluation practices.