INSANE ping spikes since the first few patches? Anyone else?

I was extremely surprised how well the game was doing since day 1 but after the first few patches it's just going downhill...
Temple of Chaos is especially bad, this thing is killing me... Delete this short freeze animation from the Trialmaster when he starts talking to you, if I Shield Charge in that exact moment, the game drags me back and the ping spikes to 300ms short before disconnecting. This is just really bad code.

I started with a constant 13ms on Opening Night without any disc for like 12hrs, but now I barely get through 5 maps without getting kicked.

Playing on a 1000 Mbit cable I doubt it's my hardware? Is it my build? some Ice skeletons with some SRS can't be the issue here... Anyone else experiencing deteriorating pings?