Hurricane Helene Affecting US East Coast Services

Tweet

This email is to address the ongoing outage affecting services at our US East Coast facility in Lenoir, NC. While there are no additional updates beyond those available on our status page, we wanted to provide more context and information regarding the incident. In our 13+ years of business, this is the first time we’ve encountered an outage of this scale.

Hurricane Helene, a Category 4 storm, made landfall on the east coast of the US on Sept. 26, 2024, at 11:10 p.m. EDT. The storm has caused widespread disruption across Florida, Georgia, and the Carolinas. The City of Lenoir, NC, where our East Coast facility is located, issued the following advisory:

"This will be one of the most significant weather events to affect western portions of [North Carolina] in modern times. Record flooding is expected, surpassing even the floods of 1916 in the Asheville area. The impacts are forecasted to be greater than Tropical Storm Fred in August 2021 and the 2004 events from Hurricanes Frances and Ivan."

Our data center partner at this location took extensive steps to prepare, including housing staff on-site for continuous 24x7 monitoring, support and ensuring ample fuel for the backup generators to provide power in case of a power outage. The facility also has diverse network feeds to ensure connectivity during such events.

However, the first network feed went offline shortly after the storm hit, and traffic was rerouted to the second network feed, which subsequently went down at Sep 27, 2024 - 08:39 EDT as well. This resulted in a complete loss of network connectivity for the entire facility, affecting us and other tenants at the data center.

While the facility itself isn't affected and has had power, although it has been intermittent. The backup diesel generators have been running as needed. The high-efficiency UPS systems are also functioning as expected to provide stability during any power fluctuations. The surrounding infrastructure has been severely impacted, including widespread power outages and damage to carrier networks. Despite having multiple points of entry, external carrier infrastructure has not yet been restored due to ongoing weather-related challenges. Power and network service providers are working to restore services as quickly and safely as possible. At this time, we cannot provide an estimated time for full resolution. We understand the frustration and disruption this causes to your business, and we deeply empathize.

What Are We Doing Now?

While we await network restoration, we are offering affected clients the option to provision services at one of our alternate data center locations, either on the West Coast (US) or in London, UK. If you have backups available, we can also restore them to these locations at no cost. Once services are restored, you may choose to continue operating from the alternate site or return to the original East Coast location. If you'd like to pursue this option, please submit a support ticket.

Additionally, for clients located in Florida, Georgia and the Carolinas, we will ensure that your invoices do not become overdue, and no late fees, service suspensions, or terminations will occur.

Future Measures

Unfortunately, natural disasters like this are beyond human control. Despite our best efforts to prepare, edge cases such as this can overwhelm even the most robust precautions. We often receive questions about high availability and redundancy. While we do have redundant power and network feeds, the severity of this event took one of them (network) offline. Most consumer grade "cloud"/"high availability" solutions involve a systems/server level high availability, that is if one server or a drive goes offline, the service fallbacks to another. The only kind of setup that would help in the kind of scenario we are experiencing right now would be the geographical redundancy - which involves services provided out of redundant physical geographical location and while this is technically possible it would be prohibitively expensive for the kind of services we (and most other hosting companies) provide and would be an entirely different market segment to cater to. It simply is not possible to do this at the $20 - $100 price point.

However, one area we will improve upon is our backup strategy. Currently, our backups are stored in the same facility as the services they support (West Coast backups in West Coast data centers, East Coast backups in East Coast facilities, etc.). While this setup allows for faster backups and lower costs, this incident has demonstrated the need for offsite backups. Going forward, we plan to maintain a less frequent monthly backup at a remote offsite location. This will enable us to restore services to a different location, though the process may be slower. We aim to implement this soon.

Finally, our thoughts and prayers are with everyone impacted by this catastrophic event, and we deeply appreciate the tireless efforts of those working to restore power and essential services.

Please continue to monitor our status page (https://backyard.host4geeks.com/serverstatus.php) for the latest updates.

Thank you for your patience and understanding.