Hayoxi's fulmination doesn't always trigger with Despair

As title says, even with all conditions applied, it sometimes doesn't trigger. I have tried to figure out why, testing a lot with different setups of support gems and different cast speeds. I have nothing in my tree saying that duration of curses are longer or shorter so it should be base speed of curses. So there seems to be no rhyme or reason to why it doesn't trigger. I have it on Despair with my Chaos Lich build. This is quite annoying since I seem to be the only one in the world paying with this support so no one else is having this issue :D
The payoff comes after 10+ seconds so it is quite important that it actually triggers.

I can provide a video if you like.

Found something, it seems to be something to do with weapon swapping. So I cast either Essence Drain or Chaos Bolt with my primary weapon and then cast Despair which is on my secondary weapon it won't trigger if I then swap back to my primary. (It does work if I keep my secondary out but that is no good, then there's no point of swapping) I have made sure that I have nothing in the passive tree affecting curses, durations or whatever. It still breaks on weapon swap. The weird part is, that it DOES sometimes work but difficult to show since it is me just noticing when playing that it does sometimes work.

So I do think this is an unintended reaction happening cause of weapon swap?

Here's a video showing it in the hideout