Dudley was born in Stamford, Connecticut, the son of Elizabeth Josephine (née Kovacs, c. 1941– ), a teacher, and Guilford Dudley III (c. 1932– ), a minister. His maternal grandfather, also a minister, immigrated from Hungary, and his maternal grandmother's parents were also Hungarian. His paternal grandfather was Guilford Dudley, who was U. S. ambassador to Denmark under the Nixon and Ford presidential administrations.