Background
The evaluation of the Robert Wood Johnson Foundation’s (RWJF) Healthy Kids, Healthy Communities (HKHC) national program is an example of an effort to apply systems science and mixed-methods evaluation approaches to comprehensive policy, systems, and environmental interventions. The HKHC national program (www.healthykidshealthycommunities.org) supported community-based efforts to implement policy, system, and environmental changes aimed to make communities healthier, particularly for higher-risk children and families (ethnic/racial minorities, lower-income populations, or those living in southern states), by increasing both active living and healthy eating. RWJF funded one year of evaluation planning (mid-March 2009 to mid-March 2010) and four subsequent years to support a mixed-methods evaluation of HKHC (April 2010 to March 2014), including all 49 communities across the United States and Puerto Rico. Communities selected to participate in this multi-year demonstration varied in population and geographic sizes (municipal to eight counties), sociodemographic composition (median annual household income, race/ethnicity, urban/suburban/rural), scale (county-wide to specific organizations or settings), scope of their proposed strategies (e.g., new or modified parks versus nutrition assistance in farmers’ markets), lead organizations (nonprofit, education, philanthropy, government), and age of the community partnerships. The evaluation did not focus on changes in individual behaviors and health outcomes.
Description
Eight complementary evaluation methods addressed four primary aims seeking to: 1) coordinate data collection for the evaluation through the web-based project management system and provide training and technical assistance for use of this system; 2) guide data collection and analysis through use of the Assessment & Evaluation Toolkit; 3) conduct a quantitative cross-site impact evaluation among a subset of community partnership sites; and 4) conduct a qualitative cross-site process and impact evaluation among all 49 community partnership sites. The evaluation consisted of the following key components: HKHC Community Dashboard: This web-based project management system (www.hkhcdashboard.org) coordinated data collection for the evaluation. It was designed to encourage the formation of a collective learning network among community partnerships, Project Officers, and Evaluation Officers. This website included functions such as social networking, progress reporting, and access to the assessment and evaluation toolkit to maintain a steady flow of users over time and increase peer engagement across communities. Individual and Group Interviews: Evaluators collaborated with community partnerships to conduct individual and group interviews with staff, partners, and community representatives before, during, and after site visits. Interview protocols focused on organizational and community factors influencing processes and means used to develop, implement, and enforce policies. In addition, evaluators tracked costs and funding associated with the design, development, implementation, and enforcement of cross-site strategies. Group Model Building: The evaluation team and partners from the Social System Design Lab at Washington University in St. Louis co-designed a group model building process to develop behavior-over-time-graphs and graphical system dynamics models (causal loop diagrams) with community partnerships. These exercises provide deeper and shared insights among representatives from the community partnerships into the drivers of obesity dynamics, better understanding of local systems at play, more rigorous critique of assumptions underlying the systems, and greater “buy in” to high-leverage prevention policy recommendations. Enhanced Evaluation: The evaluation team created tools, protocols, and trainings for environmental audits and direct observations associated with cross-site strategies to be conducted by community partnerships. Participation in these methods was voluntary, yet 31 of 49 community partnerships engaged in these activities. Supplemental Methods: Evaluators also collected and analyzed data from an online partnership and community capacity survey, photos, community partnerships’ annual narrative and financial reports, and surveillance systems (e.g., U.S. census). A synopsis of cross-site findings with community examples will be presented.
Lessons Learned
Several themes emerged, including: the value of systems approaches, the need for capacity building for evaluation, the value of focusing on upstream and downstream outcomes, and the importance of practical approaches for dissemination. Constraints included: a lack of standards in the field for indicators and measures of many of these factors, difficulty in attributing effects or impacts to specific strategies, and challenges with analyzing, interpreting, and applying what is learned, particularly with respect to complex systems science methods.
Conclusions
Community-based initiatives such as HKHC provide promising approaches for addressing childhood obesity. This presentation illustrates how mixed-methods evaluation approaches can provide practice-relevant evidence that has the potential to improve population health. The mixed-methods evaluation of HKHC advances evaluation science related to community-based efforts for addressing childhood obesity in complex community settings.
Next Steps
This evaluation will inform research and practice related to the design, implementation, and evaluation of policy, system, and environmental interventions; key partners to engage in the process to change community environments; and possible causal relationships among social determinants as well as factors associated with partnership and community capacity that influence healthy eating and active living policies and environments, and health and health behaviors.
References
Evaluation of Healthy Kids, Healthy Communities Supplement to be published in March/April 2015.
Support / Funding Source
Support for this evaluation was provided by a grant from the Robert Wood Johnson Foundation (#67099).
