Weapon Set Skill gem passives

Hello,
I have found the weapon set skills gem, with passives swapping clunky.
As you may have seen, many build guides suggest hard swapping passives across weapon swaps.
This feels bad design to not allow us to use the same passive gem in two skills which are active in different weapon sets. Please could this be changed to allow more freedom and fluidity between the two sets ? making me press one button to go from a boss build, and a mapping build ?

Greatly appreciated.