Neglectful rearing is linked with young adults’ substance use and abuse, though the developmental mechanisms that underlie this association are unclear. The present study examines links between self-esteem growth during adolescence, childhood supervisory versus physical neglect severity, and substance use and abuse in young adulthood. A sample of youth was obtained from the Add Health study (N = 8738; 55.4 %-Female; 20 %-African American, 14.7 %-Hispanic). Growth mixture modeling analyses supported declining, ascending, and stable high self-esteem trajectories. The declining and ascending trajectories reported greater neglect and alcohol abuse (but not use) as well as cannabis use and abuse. The findings suggest that compromised development of self-esteem underlies associations between neglect and substance use and abuse. Preventive interventions may benefit from targeting self-esteem among neglected youth.