Background:
Identification of effective strategies is critical to build the evidence base for community level obesity prevention. Rigorous evaluation and empirical research studies are costly and time-consuming. In addition, even well-designed studies may be unable to reach a definitive conclusion because of unanticipated barriers such as lack of full implementation, inadequacy of data systems, or significant shifts in staff understanding of goals for a program or policy. Evaluability assessments provide a systematic way to determine if the program or policy is ready for rigorous evaluation and identifies what refinements may be needed prior to a formal evaluation study. Evaluability assessments can also determine what potential barriers an evaluation study might need to address. In contrast to typical evaluability assessments, this project conducted evaluability assessments on groups of programs/policies in different topical areas.
Objectives:
To conduct evaluability assessments on groups of selected programs and policies that have been implemented to improve eating habits and physical activity levels to prevent childhood obesity.
Methods:
Evaluability assessments were conducted on groups of programs and policies in three themed areas during the first year of the project: local wellness policies in schools, afterschool/daycare programs or policies, and access to fresh foods in inner city, low income communities. Priority was given to programs and policies that addressed low income populations and ethnic groups that experience disproportionate childhood obesity. A systematic search process identified nominee programs and policies, which were selected for an evaluability assessment based on criteria developed in advance including potential impact, innovativeness, reach to target population, acceptability to stakeholders, feasibility of implementation, feasibility of adoption, transportability, sustainability, and organizational capacity.
Evaluability assessments were conducted to answer the following questions: Is it plausible that the intervention will produce the desired outcomes, leading to the prevention of childhood obesity in the target population?; Is it feasible that the program/policy will be fully implemented as intended?; and What are options for further evaluation? Each evaluability assessment consisted of site visitors reviewing program or policy documents, traveling to the program/policy location to assess implementation, data collection, staff capacity, and intended outcomes, and providing feedback and assessment for its readiness for comprehensive evaluation. A two-person team conducted each two and a half-day site visit.
Each team conducted approximately 9 to 12 interviews with key informants to gather information about the plausibility and feasibility of each program or policy. Site visitors developed a logic model prior to the visit to facilitate interviews (particularly with the lead administrator and managers). Structured observations of the sites and environmental assessment checklist were completed. The site visitors concluded the visit by reviewing and refining the initial logic model with the lead administrator and program or policy stakeholders. This discussion included technical assistance (TA) to the program or policy regarding its logic and plausibility of accomplishing its intended goals. After each site visit the site visitors prepared a written assessment for the program or policy as well as supplemented the initial TA by a brief follow-up TA telephone call with the program or policy stakeholders.
Results:
In Summer 2007, a panel comprised of 15 national experts in nutrition, physical activity, and evaluation reviewed and selected 26 programs and policies among the three themes to receive evaluability assessments. Selection included 6 local school wellness policies, 10 afterschool/daycare programs, and 10 access to fresh foods programs. In Spring 2008, 30 programs and policies will be identified by the expert panel for three new or expanded thematic areas. Synthesis reports for each theme will be written based on the findings from the evaluability assessments. These reports will identify programs/policies ready for full evaluation.
Conclusions:
This project demonstrates the feasibility of using a systematic process to conduct evaluability assessments on multiple programs/policies in a themed area to more efficiently identify programs/policies ready for a full evaluation. Procedures such as on-line ratings by expert panel members provided a systematic approach and an effective way to leverage their subject matter expertise without engaging them as site visitors. Structured protocols for site visitors allowed effective deployment of site visit teams and rapid turn-around of summary reports and conclusions about evaluability and readiness for rigorous evaluation. In conclusion, this process demonstrates that evaluability assessments are a method for identifying numerous programs and policies in a particular substantive area that are ready for a full evaluation.
Disclaimer: The findings and conclusions in this report are those of the authors and do not necessarily represent the views of the Centers for Disease Control and Prevention.
Support:
Robert Wood Johnson Foundation, the Centers for Disease Control and Prevention (CDC) and the CDC Foundation, with a contract with Macro International Inc. MOA#: 12642-0100-06 SC.
