Ashby was born in Atlantic Beach, Florida, the son of Eleanor (Johnson), a civic organizer, and Clarence Linden Garnett Ashby Jr. , a pharmaceuticals manufacturer. Ashby graduated from The Bolles School, a private school located in Jacksonville, Florida. He attended Fort Lewis College in Durango, Colorado, but dropped out his junior year to pursue an acting career. Ashby studied acting at Neighborhood Playhouse in New York City.