Question: What if you're able to contact the totem just fine, but you have a hard time working with the energy or dislike it? Is that animal off-limits as well?

Nope, I have and will continue to work with animals that are 'difficult' for me, but where the animal still contacts me or will work with me, y'know? :)