That area of the mansion is horrible. I ended up hating those giant hands, they are easily more annoying than some boss. Every time I saw them I would do the same thing, avoid them and try to advance without fighting haha.