spirit limited by weapon swap?

Don't know what caused it, i'm using a weapon swap with a scepter on one set and focus on the other, my scepter set seems to require to have enough spirit in the other set (for whatever reason this even affects the skeleton warrior provided by rattling scepter so i can only summon the 2 free skeletons with a rattling scepter in hand)

minion gems even bug out and don't summon anything if i don't have to same amount of skeletons summoned in both sets. im sitting on 200/350 spirit and my unearth build is bricked since i only have 5 skeletons now. :C