Cloud 2015 is off to a stuttering start with Verizon’s announcement of up to 48 hours of downtime for maintenance. Notifications have been going out to all Verizon Cloud customers with predictably unhappy responses. While the twitterverse is in a bit of an uproar, Verizon has performed similar maintenance in the past and states that ONLY 24 hours of downtime is expected.
Per The Register:
“We are conducting regularly scheduled system maintenance on Verizon Cloud next weekend. We’re adding several new features to the platform that will continue to improve the service for our enterprise customers. Updates of this nature typically require some system downtown and we have notified customers in advance to make sure they can plan accordingly. This is a standard practice in the industry. No customers in Australia will be impacted. We anticipate the update will take far less than 48 hours. Previous similar updates took about 24 hours.”
Update 12 January 2015: Verizon’s planned maintenance for its Enterprise Cloud was completed over the weekend. Total downtime was 40 hours. The Verizon press release that announced restoration of services states “adding seamless upgrade functionality as well as other customer-facing updates”. Let’s hope they are accurate on that. While this iteration of cloud is fairly new for Verizon (launched last fall), there is little tolerance in the crowded cloudspace for downtime. People are watching and tweeting.
We thought it might be a good time to review some of the more interesting cloud closures of 2014.
Dopbox File Sharing Fail, January 2014: A scripting glitch caused OS upgrades to be applied to active machines during a routine maintenance. Engineers were forced to restore from backup which took 2 days due to the massive size of Dropbox’s databases.
Basecamp Closes Tent, March 2014: Project management service Basecamp suffered a DDoS attack that took it offline for 2 hours. The attackers demanded money, but Basecamp refused to give in. The attack was a combination of SYN flood, DNS reflection, ICMPflooding, and NTP amplification with a combined flow in excess of 20Gbps.
Adobe’s Creativity Goes Down, May 2014: Over a million paying users of Adobe’s Creative Cloud and some secondary services were offline for 28 hours due to an issue an issue during database maintenance activity.
Evernote Uptime not Forever, June 2014: A DDoS attack took news aggregator Feedly and online note service, Evernote, offline for 10 hours. The perpetrators demanded money but neither service complied.
Xen Bug Causes AWS Reboots, September 2014: A previously unreported bug in the Xen hypervisor caused three days of rolling reboots to apply patches to 10% of AWS’s servers. The rest of the Xen-based cloud world performed similar updates including Rackspace and IBM/Softlayer. RightScale has a good write-up on the activity and its effects.
Infinite Loop Update Brings Down Azure, November 2014: Information Week indicates that Microsoft rolled out an update to its Azure storage service that contained an unintended infinite loop for a certain operation buried in the code which caused the service to freeze. Azure customers and a number of Microsoft properties; Office 365, Xbox Live, MSN were affected to varying degree. Most services were degraded for 4-12 hours.
Gamers Get Coal for Christmas, December 2014: And, of course, the most recent outages to the PlayStation Network and Xbox Live, brought on by hackers, caused a great deal of angst Christmas morning.
What to expect in 2015?
More outages. 2014 showed us that even the largest providers are susceptible to poor planning, poor engineering, unknown bugs, and malicious miscreants. While the technical issues can be prevented, we expect to see an increase in hacker caused outages as the tools to initiate these cyberattacks become widespread and they become a preferred method of social protest and nation state domination.
