Attrition's stacks/time should persist between weapon swap

If I have two weapon sets that both could utilize Attrition (persistent skill buff) then swapping between weapons shouldn't reset the timer on attrition. But I can hear the skill going off each time switch, and when testing on a boss the dps doesn't go up on attacks unless I don't swap between weapons.