Last week, Bloodlines experienced some issues related to our move to a new hosting environment. We wanted to take the opportunity to explain what happened, why it happened, and what we’ve done to resolve it.
On Friday, April 7, we sent out group notices and put out a blog post warning everyone with very short notice that the Bloodlines system was going to go down temporarily for scheduled maintenance. The reason for this was so we could re-point our server side systems to a new hosting environment that we had been preparing. We expected to be able to complete the changeover in about 15 minutes, which was about right for the majority of our systems, including the vendor system, and the application that interfaces with Second Life. However, we experienced an unexpected problem with the Bloodlines website that took us several hours to resolve so we could bring the website up again.
The Curse, which reduces metrics every night at midnight, was canceled for the night of 4/7, because of the maintenance, and was resumed the following night, Saturday. The Curse routine was running on our new host in a new environment on Saturday night, and another unexpected issue caused it to be retried 10 times, which means that many people were destroyed, lost souls, and lost 10x as much blood and lumens as they should have. As soon as this problem was discovered on Saturday morning, all destructions were immediately reversed, everyone who had lost souls as a result of this had them returned, and we ran routines to restore all lost blood and lumens to everyone who had lost them the night before. We realized today that while these routines restored metrics to the majority of players who lost them, they missed approximately 1000 players. Those players had blood and lumens restored today.
We were only able to restore blood and lumens directly to the member accounts of those who had enough room to hold the extra metrics. Anyone who didn’t have enough room in their veins to hold the restored metrics had them returned to their reserve accounts. Reserve accounts can be accessed by filling the metrics into the container at any refill station.
For many accounts, we were not able to restore the exact metrics that may have been had before the Curse error, so in some cases were erred on the side of returning too many metrics rather than too few. So if you find yourself with slightly extra that you had before the error, consider this a gift from us in return for the hassle!
Our new hosting environment is vastly more robust than our previous environment, and will allow us to recover from problems much more quickly, and to automatically scale the capacity of our systems in response to demand. The quick translation is that the game will run faster, and we’ll have far fewer outages!
In conclusion, we think we’ve resolved everything from the transition – however if you believe that you are still missing any stats, souls, or metrics as a result of these problems, or if you have any questions, please create a ticket at slbloodlines.com/support, and we will get back to you as soon as possible.
Thanks!
