Background
There is general consensus in the public health and behavioral medicine literatures around the need to validate new measures of environmental features associated with health behaviors. Selected authors (Diez-Roux, 2001; Mâsse, Dassa, Gauvin, et al., 2002; Raudenbush & Sampson, 1999) have argued that extensions to the classic psychometric methodology are required and have developed a new approach called ecometrics. Ecometrics refers to the scientific assessment of settings or environments through systematic social observation and analysis of resulting data through multilevel modeling procedures, which build on itemresponse theory and generalizability theory.
Objectives
The purpose of this presentation is to examine the validity and reliability of an environmental measure of neighborhood walkability (i.e., an emergent property of the environment that is a function of user-friendliness, safety, number/variety of destinations, and aesthetic appeal of the environment) through application of the ecometrics approach.
Methods
Data for this presentation were yielded from a larger project dealing with walkability of urban neighborhood environments. Using an 18-item observation grid, four pairs of observers (total n=8) performed field observations of 112 neighborhoods on the island of Montreal. Following a three-day training session, observers were provided with a map of each of neighborhoods, which included a pre-determined walking route constructed through the joining of 10 randomly selected street blocks. Upon arrival in the neighborhood, observers commenced their evaluations at opposite ends of the walking route and recorded their overall neighborhood ratings at the end of their walking route. Data collected in this manner produced a hierarchically structured data set including 4032 observations nested within observers, which in turn were nested within neighborhoods (18 items X two observers per neighborhood X 112 neighborhoods). Data from the 2001 census (average family income, proportion of persons without a high school education, and proportion of persons walking to work) were linked to neighborhood data.
Results
Application of ecometric multilevel modeling analyses showed that only between one percent and 14% of the variance in outcome measures was associated with between-observer variability suggesting that inter-rater differences accounted for only small amounts of variance. Once inter-item and inter-observer variability were statistically controlled, about 21% to 37% of the variability in outcome measures were at the between neighborhood level. Reliability estimates were .78 for items measuring user-friendliness, .62 for aesthetic appeal of stimuli, .76 for safety, and .82 for number/variety of destinations. While aesthetic appeal and safety of the environment were positively associated with neighborhood affluence (higher average income associated with greater aesthetic appeal and greater safety) user-friendliness and number/variety of destinations were negatively associated with affluence (greater average family income associated with lower user-friendliness and number/variety of destinations). Greater number and variety of destinations was associated with higher proportions of persons walking to work in the neighborhood suggesting good concurrent validity.
Conclusions
We conclude that the four subscales of the walkability measure have good reliability and are able to capture between neighborhood differences in dimensions of walkability. Measurement characteristics of this observational measure would have been difficult to ascertain without the ecometrics methodology.
