I think largely because it tackled a subject matter that was wildly out there for that era. And, it treated the "freaks" with respect and made the "normal" woman the villain. The director, Tod Browning, who even more famously directed the original "Dracula" with Bela Legosi, based "Freaks" on his own experiences in the circus.

Plus, it's just a darn good revenge story.

Some of the imagery and the scares still can disturb a lot of jaded horror fans (not in your case, I guess). It was very raw and real at a time when horror movies tended towards the cartoonish, like those classic Universal monsters. Dracula, the Wolfman, Frankenstein.