Built environment characteristics have been linked to health outcomes and health disparities. However, the effects of an environment on behavior may depend on human perception, interpretation, motivation, and other forms of human agency. We draw on epidemiological and ethical concepts to articulate a critique of research on the built environment and physical activity. We identify problematic assumptions and enumerate both scientific and ethical reasons to incorporate subjective perspectives and public engagement strategies into built environment research and interventions. We maintain that taking agency seriously is essential to the pursuit of health equity and the broader demands of social justice in public health, an important consideration as studies of the built environment and physical activity increasingly focus on socially disadvantaged communities. Attention to how people understand their environment and navigate competing demands can improve the scientific value of ongoing efforts to promote active living and health, while also better fulfilling our ethical obligations to the individuals and communities whose health we strive to protect.