What's going on with the Servers?

For the last six or so hours, we've been having some quite bad server issues. This manifests in several ways: some players getting disconnected from time to time, various actions serverside timing out, and rarely, items being duplicated or destroyed. This news post explains what we know about the situation and our timelines for fixing it.

This release has seen 11% higher peak player numbers than any previous Path of Exile release, and that would likely be even higher without the server issues. While this would normally be something to celebrate, right now it's causing some big server problems.

Path of Exile's server architecture consists of a central set of database servers and an arbitrarily large pool of game servers. We can add more game servers easily (and have to vary that number constantly to take into account player numbers), but the database servers are much harder to change.

For the first time since 2013, our database servers are overloaded. This is manifesting as various database actions occasionally failing (which can include player saves). While it's still possible to play normally 99% of the time, that 1% where it fails is really annoying for players right now and we're very sorry about that. We have load-tested with bots to a massive scale before and didn't run into this issue, so that's certainly going to prompt some investigation.

The save failures can also, in some rare circumstances (that can't be controlled by players), duplicate items. We don't need reports about this because we understand the mechanism behind it and it isn't intentionally abusable. It's definitely concerning but will stop happening once the server problems are resolved soon. Technically there's a 50% chance it deletes the item and a 50% chance that it creates a duplicate, and players have no control over when this occurs. This has only happened a few times so far and does not warrant an economic rollback (as scary as it sounds at first glance).

We're not asking for help with the database problem - I'm sure there are many talented database architects in the community who have advice for us based on my quick explanation. We do know what to do, it's just going to take some time. In the meantime, we're ordering some hardware upgrades and plan to put this in place as soon as possible. This may involve a quick realm restart or some downtime, and we'll keep you posted with updates to this thread as we get a better idea of timing.

We're going to delay the release of the Warlord Mystery Box (which would normally be taking place right now) until tomorrow. Thanks again for your patience. Aside from this terrible server problem, this has been our favourite launch so far and we can't wait for things to be smooth again.