Post-apocalyptic survival MMO Last Oasis is finally back online after a week of downtime following its extremely rocky Steam early access launch back in March – and developer Donkey Crew has shared a few words on what, exactly, went wrong.
Last Oasis, if you're unfamiliar, casts players as human survivors, doomed to endlessly roam an Earth that no longer rotates in order to stay ahead of the scorching sun.
It's a wonderfully appealing premise, and one that plenty of players were eager to experience for themselves when Lost Oasis entered early access on 26th March. Sadly, however, Last Oasis' launch was plagued by server issues, with severe wait times and server crashes leaving many that purchased the game simply unable play. After a few days with no improvement, Donkey Crew elected to take its servers offline for a week to get to the bottom of things.
"Our coders have been working day and night on this issue and need some sleep," it wrote at the time, "We need to properly investigate why our load testing didn't pick this issue up, find out why, figure it out properly and solve it." Donkey Crew also offered a "no questions asked" refund for those that didn't want to wait out the downtime.
Servers finally went back up for public testing this weekend, with the goal of assessing server performance following a week of fixes, and Donkey Crew also used the occasion to shed more light on Last Oasis' troubled launch, which occurred despite a year-long beta and many large-scale load tests, according to the developer.
Following a relatively smooth first few hours, it explained, things quickly began to deteriorate as "more and more people failed to connect and a large number of players were stuck in queues". Making matters worse, servers then started repeatedly shutting down.
As Donkey Crew put it, Last Oasis' lobby and joining queues simply weren't optimised enough for the "tens of thousands of players connecting simultaneously" at launch. "Clients were sending massive numbers of requests to our backend to check for the status, essentially spamming it. Doing so shouldn't have overloaded the gameservers with the verification process, but the sheer amount of those requests did." Worse, a bug meant that "clients were continuing to send those requests indefinitely" while players remained stuck in queues."
"At some point," the developer continued, "the database became so slow with all the connections that the game servers couldn't verify their consistency state on time anymore, forcing them to all shut down simultaneously. Again, it was a failsafe to make sure that the world doesn't become inconsistent and breaks the logic of the game in the long run."
"Our systems were essentially stuck in a loop with multiple issues affecting each other," explained Donkey Crew, "As all the servers were shutting down and restarting, over 20k people were trying to rejoin at the same time, leading to our queue system failing, which then kept overloading the master server, letting only a few people to join a time until the master server would shut off again and take all servers down with it. And the cycle would continue."
Donkey Crew described its attempts to simultaneously bring servers back up and fix the underlying causes as feeling like "rebuilding a house of cards in the middle of a hurricane". Additionally, the pressure and stress of the occasion "mixed with worsened communication due to the quarantine situation made things even more difficult."
After "days of working non-stop", Donkey Crew made the decision to take its servers offline and get some rest before attempting to tackle the issues again.
The next day, though, it "woke up to something we did not expect: a lot of you expressing understanding and support. We can't tell you how much of a morale boost that was for everyone on the team. It helped us to get our shit together: organise properly, analyse logs and code and fully check what went wrong. Over the last few days, we can easily say we found a lot. We've been refactoring large parts of the code that we've seen cause problems, changing the database structure that was failing, and fixing various integrity issues that appeared. As far as we can tell, all the known issues have been fixed."
That optimism appears to have held true; Saturday's public test went well enough for Donkey Crew to leave its servers up, with no additional wipes or server shut downs required, and Last Oasis has remained operational since then with minimal disruption.
Hopefully, with server issues seemingly resolved, Donkey Crew can now put Last Oasis' rough launch behind it and start building on the game's early promise, as highlighted by Eurogamer's Emma Kent when she took an pre-release look at the game. The developer hopes to get the ball rolling with a new content update due later this week.
