Spirit bug?

Every time I zone it says I don't have enough spirit for my minions. Desummons them all. I have to hit the minus on all of them, then my companion will spawn. Then I have to unbind my spectre, put in inventory, requip it without supports, then resummon all my minions and put back in supports. Then it works. But everytime I zone it does this. If I do not put my spectre in my inventory it lets me hit the plus button infinitely but says I have no spirit. It will also not let me summon any minions until I take my spectre out and put in inventory.