Exploring the Potential of Design-Driven Evaluation

Abstract Information: Design-Driven Evaluation (DDE) is a participatory evaluation model that uses design methods to evaluate and create a newly designed product, such a policy or program (Norman, 2021). Norman hypothesizes that DDE is particularly suited to driving systems change in complex social environments because of its union of evaluation and innovation. This paper focuses on our use of DDE with a multi-year STEM education initiative that uses professional development to change teacher knowledge, beliefs, and behavior. The initiative also produces curriculum that can be used by teachers across our state. The paper focuses on our application of a co-design approach, thus extending Norman’s DDE model; what we are learning about how evaluation teams can employ design-based methods in evaluation; and our use of DDE within a larger evaluation framework. Primarily, we discuss how we have extended the DDE model to include a co-design approach. While co-design has been used in other participatory evaluation methods, like Developmental Evaluation (Lam & Shulha, 2015), it was not central to Norman’s DDE model. We share practical ways that co-design can be included, drawing from the Design-Based Research (DBR) literature and our own experiences. DBR has been used since the 1980s in the field of education (Barab, 2005). When co-design is a part of the DBR model, it can contribute to individual and organizational learning (Collins, et al., 2009; Penuel, 2011). Co-designed DBR can increase participant knowledge and confidence about specific content areas (Fragat, et al., 2022; Annetta et al, 2013; Swan, 2007) and their own learning process (Kali et al., 2018). Through the active participation of diverse perspectives, co-designed products can also be more culturally responsive (Belgrave et al., 2022) or increase participant critical consciousness (Penuel et al., 2023). In our own evaluation, we are using a co-design approach that includes instructional coaches, evaluators, and teachers. We aim to cultivate sustained capacity among teacher participants to design new STEM curriculum and integrate novel instructional approaches, including culturally responsive pedagogy. We will discuss what we are learning about how to use the co-design process effectively. DDE may seem daunting to evaluation teams that do not have design training. We will discuss what we have learned about the processes, roles, dispositions, and knowledge that are needed to employ a DDE model. We will also discuss how the evaluation design has helped our team adapt to a complex and evolving evaluation context. Most DBR is intensive and conducted over a short period of time, typically ranging from several months to one year. However, this does not suit many evaluation designs that unfold over long periods of time within natural environments. We will discuss how we have incorporated DDE not as an intensive or all-encompassing aspect of our research, but rather as part of a larger evaluation framework that includes an impact evaluation. We hope that our approach can be instructive to other evaluators and help increase the types of environments where DDE might be used.