Rackspace hosted Trade suffered a catastrophic outage starting December 2, 2022 and remains to be ongoing as of 12:37 AM December 4th. Initially described as connectivity and login points, the steerage was finally up to date to announce that they had been coping with a safety incident.
Rackspace Hosted Trade Points
The Rackspace system went down within the early morning hours of December 2, 2022. Initially there was no phrase from Rackspace about what the issue was, a lot much less an ETA of when it might be resolved.
Prospects on Twitter reported that Rackspace was not responding to assist emails.
This has been fairly the day with #Rackspace. Each hosted change consumer has been down for 14 hours or so. Assist is not studying/responding to tickets. Updates are unhelpful.
I’m involved now that they fell sufferer to one thing dangerous just like the ProxyNotShell PoC hack. https://t.co/jchKsAO3Z7
— Joe Sinkwitz (@CygnusSEO) December 2, 2022
A Rackspace buyer privately messaged me over social media on Friday to narrate their expertise:
“All hosted Trade purchasers down over the previous 16 hours.
Undecided what number of firms that’s, however it’s important.
They’re serving a 554 lengthy delay bounce so individuals emailing in aren’t conscious of the bounce for a number of hours.”
The official Rackspace standing web page provided a working replace of the outage however the preliminary posts had no info aside from there was an outage and it was being investigated.
The primary official update was on December 2nd at 2:49 AM:
“We’re investigating a problem that has effects on our Hosted Trade environments. Extra particulars shall be posted as they change into accessible.”
13 minutes later Rackspace started calling it a “connectivity concern.”
“We’re investigating reviews of connectivity points to our Trade environments.
Customers could expertise an error upon accessing the Outlook Internet App (Webmail) and syncing their e-mail consumer(s).”
By 6:36 AM the Rackspace updates described the continued drawback as “connectivity and login points” then later that afternoon at 1:54 PM Rackspace introduced they had been nonetheless within the “investigation section” of the outage, nonetheless attempting to determine what went fallacious.
And they were still calling it “connectivity and login points” of their Cloud Workplace environments at 4:51 PM that afternoon.
Rackspace Recommends Migrating to Microsoft 365
4 hours later Rackspace referred to the scenario as a “important failure”and started providing their clients free Microsoft Trade Plan 1 licenses on Microsoft 365 as a workaround till they understood the issue and will convey the system again on-line.
The official steerage acknowledged:
“We skilled a big failure in our Hosted Trade setting. We proactively shut down the setting to keep away from any additional points whereas we proceed work to revive service. As we proceed to work by the basis explanation for the difficulty, we have now an alternate answer that can re-activate your capacity to ship and obtain emails.
For free of charge to you, we shall be offering you entry to Microsoft Trade Plan 1 licenses on Microsoft 365 till additional discover.”
Rackspace Hosted Trade Safety Incident
It was not till practically 24 hours later at 1:57 AM on December third that Rackspace formally introduced that their hosted Trade service was affected by a safety incident.
The announcement additional revealed that the Rackspace technicians had powered down and disconnected the Trade setting.
“After additional evaluation, we have now decided that this can be a safety incident.
The identified influence is remoted to a portion of our Hosted Trade platform. We’re taking vital actions to judge and defend our environments.”
Twelve hours later that afternoon they up to date the standing web page with extra info that their safety group and outdoors consultants had been nonetheless engaged on fixing the outage.
Was Rackspace Service Affected by a Vulnerability?
Rackspace has not launched particulars of the safety occasion.
A safety occasion usually includes a vulnerability and there are two extreme vulnerabilities presently within the wile that had been patched in November 2022.
These are the 2 most present vulnerabilities:
Microsoft Trade Server Server-Facet Request Forgery (SSRF) Vulnerability
A Server Facet Request Forgery (SSRF) assault permits a hacker to learn and alter information on the server.
Microsoft Trade Server Distant Code Execution Vulnerability
A Distant Code Execution Vulnerability is one wherein an attacker is ready to run malicious code on a server.
An advisory published in October 2022 described the influence of the vulnerabilities:
“An authenticated distant attacker can carry out SSRF assaults to escalate privileges and execute arbtirary PowerShell code on susceptible Microsoft Trade servers.
Because the assault is focused towards Microsoft Trade Mailbox server, the attacker can probably acquire entry to different assets by way of lateral motion into Trade and Lively Listing environments.”
The Rackspace outage updates haven’t indicated what the precise drawback was, solely that it was a safety incident.
Essentially the most present standing replace as of December 4th acknowledged that the service remains to be down and clients are inspired emigrate to the Microsoft 365 service.
Rackspace posted the following on December 4, 2022 at 12:37 AM:
“We proceed to make progress in addressing the incident. The provision of your service and safety of your information is of excessive significance.
We now have dedicated intensive inner assets and engaged world-class exterior experience in our efforts to attenuate adverse impacts to clients.”
It’s attainable that the above famous vulnerabilities are associated to the safety incident affecting the Rackspace Hosted Trade service.
There was no announcement of whether or not buyer info has been compromised. This occasion remains to be ongoing.
Featured picture by Shutterstock/Orn Rin