Can't upgrade skills when using Jarngreipr Ringmail Gauntlets

Despite the fact that I have the necessary level and stats to upgrade my skill gems, I can't upgrade my skills when using the Jarngreipr unique gauntlets. I have to change characters to a huntress to upgrade my skill gems and then swap back to my warrior to use them. This is very annoying.