Isabelle Adjani in Possession is the biggest pure horror performance to get snubbed the Oscars imo

There are others like Nicholson and Duvall in The Shining, fun fact Shining was so hated at the time that it received Two Razzies one for Duvall and Kubrick!

Sophia Wilde- Talk to Me
James McAvoy- Split
Tilda Swinton- Suspiria
Jeff Goldblum - The Fly

Could be other shouts