On Saturday the 21st of November, we performed scheduled network maintenance: we transitioned to a new network cluster that is better prepared to handle larger amounts of traffic, and will allow us to provide a more resilient and stable service.
Unfortunately, during our maintenance window, we encountered a serious complication with our networked storage cluster. This has lead to ongoing downtime on some of our nodes. Our 24/7 team of technicians has been working tirelessly to restore service.
At this time, the majority of affected customers is back online, and has been for some time. Unfortunately, a minority of nodes are still having issues. Our current hope is to have these servers restored tomorrow, but it's a work in progress and we're unable to offer an exact timeline. We are doing everything we can to restore service for the affected customers as soon as possible, and have brought in third-party specialists in the type of issues we're seeing to help us resolve this in a timely manner.
Because of the overwhelming amount of support tickets we've received regarding this issue, and because our focus is on resolving them as quickly as possible with all hands on board, our team may occasionally be slower to respond to support requests than we'd like while we resolve this incident.
A further technical update will follow in this same announcement tomorrow. Please let us know if you have any questions or concerns. Once again, we deeply apologize to the customers who are still affected by this for the inconvenience and continued downtime, and are doing everything we can do bring affected servers back online.
Update - November 26th
We've brought a large number of server back online today, and stabilized a node that had already been back online, but was seeing residual issues. Unfortunately, some servers are still offline, and will require further investigation tomorrow. We appreciate your continued patience if your server has not yet come back online today, and expect to make further progress tomorrow.
Update - December 1st
We've brought almost all servers back online as of several days ago. Unfortunately, a small number of customer VPSes suffered data loss during the outage of our networked storage cluster, and could not be recovered due to extensive filesystem corruption despite our best efforts. For these customers, we are doing the following:
- If you had an offsite backup service with us, we're working to restore your server from that backup. If you've not already receive an update from us on this, you will very soon.
- If you had not purchased a offsite backup service with us, we're working to recreate your server with a fresh installation. Unfortunately, in this case, we don't have any further data we can give you, and you will need to restore your own backups if you were taking backups.
We're working on this on a case-by-case basis, depending on the data we have for you.
Wednesday, November 25, 2020