Joseph was born on the border of Kentucky and West "by God" Virginia, where the Hatfields & McCoys feuded for generations over a razorback pig. He is the son of an immigrant from Seoul, Korea and an Irish Catholic nun / former beauty queen (Miss Welch) from the coal mining hollers of West Virginia.

His father, Kim Hak-Lim, was a South Korean POW and left for dead after being marched to exhaustion by North Korean troops. His grandmother, who had secretly followed behind, found him in a cave and nursed him back to health. He had survived by eating crickets and insects. His father eventually became a surgeon and was one of the first wave of Korean doctors to immigrate to the US. During his post-residency he married Shirley Jean Daugherty after she left the convent.