Background and Purpose
Over the past decade, public and private U.S. funders have invested in research and evaluation to understand the most effective, feasible, and sustainable strategies to combat childhood obesity. This evidence is used to aid practitioners and decision-makers at the organizational or agency, community, state, or national levels in selecting strategies to best fit their health, economic, environmental, and social circumstances. Current comprehensive review systems (such as the Community Guide and the Cochrane Review) provide guidance to practitioners and decision-makers interested in implementing change; yet, keeping up with the vast amount of research and evaluation data generated in the field is an ongoing challenge. In turn, decision-makers often rely on insufficient evidence as well as reviews focused more on assessing the internal validity of study results without complementary evaluation of the external validity (e.g., reach, implementation fidelity, and sustainability) associated with intervention impacts.
Objectives
The aims of the review were to: 1) develop and apply replicable methods – modeled after respected formal systematic evidence review systems (e.g., Community Guide) – to assess the scientific and grey literature addressing policy and environmental strategies for reducing obesity levels, improving healthy eating, and/or increasing physical activity among youth aged 3-18 years of age; 2) summarize these findings using easy-to-read evidence maps that identify effects/associations related to obesity/overweight, physical activity, and nutrition/diet outcomes; and 3) classify intervention strategies, based on their effectiveness and population impact using ratings ranging from “effective” (recommended for use) to “promising” and “emerging” (recommended for further testing).
More comprehensive reviews stemming from improved reporting and review standards may provide a better platform for practitioners, decision-makers, evaluators, and researchers to understand the effectiveness and impact of interventions to prevent childhood obesity.
Methods
Investigators created a protocol to systematically identify, abstract, review, and rate evidence from a variety of sources (e.g., intervention evaluations, associational studies). The ratings were designed to reflect effectiveness (study design, intervention duration, effects or associations) and population impact (effectiveness plus potential population reach –participation or exposure and representativeness) of multicomponent and complex interventions, with a particular emphasis on impacts for racial/ethnic and lower-income populations of greatest need for these interventions. Over 2,000 documents, published between January 2000 and May 2009 in the scientific and grey literature, were identified (2008-2009) and systematically analyzed (2009-2012). Studies focused on policy or environmental strategies to reduce obesity/overweight, increase physical activity, and/or improve nutrition/diet among youth (3-18 years). Related articles (i.e., those corresponding to an intervention or associational study) were grouped together into a “study grouping.” Study groupings were categorized into one or more of 24 independent strategies to increase healthy eating or active living. Investigators used the RE-AIM framework (i.e., Reach, Effectiveness, Adoption, Implementation, and Maintenance) both to assess internal and external validity, and to derive standard, objective ratings of intervention effectiveness and impact for each study grouping. The assigned ratings were then entered into an Access database to generate reports for a range of indicators (e.g., outcomes assessed, intervention components, funding sources) within and across strategies.
Results
From 396 study groupings (600 independent articles) included in this analysis, 142 (36%) were intervention evaluations and 254 (64%) were associational studies. Reported outcomes varied, including physical activity (45%), obesity/overweight (25%), nutrition (18%), sedentary behavior (2%), and other shorter-term proxies, such as trail use or fruit and vegetable purchases (10%). Evidence for intervention effectiveness was reported in 56% of the evaluation, and 77% of the associational, study groupings. Among intervention evaluations, 49% had sufficient data for population impact ratings, and only 28% qualified for a rating of “high population impact.” Moreover, only 15% of intervention evaluations had sufficient data to provide high-risk population impact ratings, and only 9% qualified for a rating of “high” for high-risk population impact.
Conclusions
This study employed ways to build on assessments of internal validity to rate effectiveness and to evaluate external validity to rate population impact, thereby helping to characterize and synthesize practice-based evidence. Among studies eligible to receive ratings, investigators noted significant variation in methods, measures, and reporting. Other studies failed to report on key elements required for assessing the internal or external validity of intervention effects and impacts, including those elements specified by the RE-AIM framework.
Implications for Practice and Policy
This work helps to accelerate the pipeline of evidence, moving from evaluability assessments to syntheses of effectiveness and impact to rigorous expert review systems. To increase real-time evidence review and dissemination efforts, researchers and evaluators have to agree on standardized indicators and reporting mechanisms in all peer-reviewed publications. This analysis identifies several indicators that can be incorporated consistently to improve review and reporting standards, thus enhancing the ability of evaluators to assess internal and external validity. In response, these efforts can more systematically enhance the knowledge base and improve recommendations for practitioners and decision-makers interested in childhood obesity prevention in both the general population and in high-risk populations.
Support / Funding Source
Support for this study was provided by a series of grants from the Robert Wood Johnson Foundation (#63675, 65518, 67413).
