Garai was born in Hong Kong, to British parents. Her father's family is of Jewish background. Her mother, Janet A. (née Brown), brought up Romola and her three siblings. Her father, Adrian Earl Rutherford Garai (born 1945), is a bank manager. Garai's great-grandfather, Bernhard "Bert" Garai, an immigrant from Hungary, founded the Keystone Press Agency, a photographic agency and archive, in London, in the early 20th century.