Email provider Rackspace down after Security Breach.

[timeshifter]

I can't find a phone number to call BitTitan to confirm how this all works

That's why I went with SkyKick. I've only done a few of these migrations over the last couple of years. When I was trying to pick a provider BitTitan and SkyKick were the two. SkyKick offered phone support. BitTitan did not. There may be a way to call BitTitan.

 

[YeOldeStonecat]

So, most migration tools like BitTitan use a 2x sided method to migrate.
On the left side of the screen is your "Source servers"..where you enter their connection properties, as well as the usernames and passwords of the mailboxes.
On the right side of the screen is your destination servers, same info.

It's a pull...push thing.

I believe RackSpace took their Exchange servers down...offline. Without any way to connect to their OWA address...I don't see how BitTitan would work.

I see you may have had that "hybrid" setup, several im-po-tant users on Exchange, the rest of the troops on POP/IMAP mailboxes.
I'm not sure of the details of this outage, perhaps the POP/IMAP mailboxes at RS are up and functional, but I think the routing connector for the hybrid setup was in Exchange...so not sure if they're working. And without Exchange up, I think your only chance is on the clients computers...in Outlook, export entire mailbox to PST from the OST cached copy. Hopefully your account properties had the slidebar all the way to the right for "download all email".

 

[callthatgirl]

I don't think you can get any mail right now, just what is local in their Outlook/Mac Mail/etc.

 

[thecomputerguy]

Your only option right now is to setup empty mailboxes at M365, then change DNS records so that they can at least start receiving mail again into brand new empty mailboxes.

Then you can, on a one by one basis, login to each individual station and export the now non receiving profile into a PST.

Then create a new Outlook profile and connect it with the empty mailbox at M365.

Then if you choose to, you can import the PST into the empty mailbox to get whatever data you can, into this empty box. Or leave it empty and have them start fresh with the hope that Rackspaces mail servers come back online, then and only then does MigrationWiz or Skykick become relevant.

I'm not sure what issues may arise if you import a half-finished PST file into the empty mailbox then MigrationWiz it after Rackspace comes back online. Worst case scenario I suppose is some significant duplication. Best is case is it doesn't migrate duplicates.

MigrationWiz and Skykick can't do anything for you right now until their servers come back online, but, they might be offline forever.

Ideally, in a perfect world, you set your clients up at M365, and they start with a fresh empty box. Then a day or two from now Rackspace get's their servers back online, they say "HEY! WERE FIXED COME BACK TO US!", then you say no thank you, and MigrationWiz all historical data over to M365. If they don't come back online, then you import the PST and it is what it is.

Regardless, exports to PST need to happen.

 

[callthatgirl]

Apple mail is a complete nightmare, client called Apple and here is what he said to get it (so far)

We did it by exporting the mailbox. We then imported that mailbox back into MacMail, and it goes to a box on MacMail called "on this mac", and then from there we dragged those emails into my Google account and it populated online.

Weird fix but this is hopefully it will work. Cannot have 2 exchange accounts in MacMail no matter what, exporting does mbox, conversion is a bitch, can't copy email out of MacMail, won't sync. Crazy....

 

[Sky-Knight]

Only way out for these poor people right now is OST exports and imports into fresh mailboxes... and worry about getting the old mail out of the downed system if and when Rackspace gets it back online.

But considering it's a down thanks to the fact they didn't bother to patch the Exchange cluster... I wouldn't hold my breath. If there is no 3rd party mail backup on that platform, that mail is gone.

Everyone on that mess should have been off a decade ago anyway.

 

[brandonkick]

So here is where I'm at right now, after working at this several hours today.

Rackspace hosted exchange is belly up for now. Like you all said, who knows if or when that will come back on line. Thank GOD (or whoever it is you thank) that the two hosted exhange mail box users actually did use the desktop client app. Was the mail slider set to "keep everything" (all the way to the right)... I don't know. I DO know that when I just not created a brand new local outlook profile for my own email address at that company, the default setting had it all the way to the right. Regardless, those two hosted exhange users will get whatever they have in their local computer PST's at a minimum. I'm optimistic they are likely facing just the two days worth of missing emails.

I now have the domain added to the tenant and mapped out all the aliases from the old rackspace structure to the new domain setup. So joe@oldcompany.com will now become an alias for joe.lastname@company.com

I can update the MX records and effectively all the "new" mailboxes will get any mail sent to the original rackspace addresses.


I need now to figure out how to get this local PST of my mailbox from rackspace, up into my new account in the new 365 tenant.


Luckily, most users have... 100-300MB of data max. My box is about 1GB, one or two others are like 2GB - 4GB. The hosted exhange boxes... they are 50+GB

What a **** storm!

 

[brandonkick]

Ok. So I can do this "manually".

I created a PST for my mailbox after it finished syncing everything to outlook desktop client. Then signed in to the new M365 account in the same outlook desktop client. From there, file ==> Import / Export ==> Import from a file, selected the backup PST I created.

Lobbed all the mail into the new mailbox, which started instantly pushing it up to the M365 online store.


A bit tedious, but will work and will be quite seemless for the two hosted exchange users with larger mailboxes. Their desktop will seem "up to date" and fine while it chew through shoving 30-50GB of mail up the pipe.

It took around 15-20 min each way (pulling the data down from Rackspace, and pushing it back up to M365) for my 1.2GB mailbox.


My main workstations bandwith is a bit pitiful in this day and age, so they should see much better speeds. But still I figure a few min for the regular IMAP boxes, they are small. Maybe a day or two for the 30-50GB exchange boxes. If and when Rackspace exchange comes back online, just create another partial PST filtered on date sent / date recieved and import that?

 

[callthatgirl]

50 gigs can take 2 or 3 days, depending on the internet speed.

yes, if/when it comes back on, you can just add the PST to the profile and manually copy over the older data, do not move....copy. I don't trust moving anymore lol.

 

[brandonkick]

Only way out for these poor people right now is OST exports and imports into fresh mailboxes... and worry about getting the old mail out of the downed system if and when Rackspace gets it back online.

But considering it's a down thanks to the fact they didn't bother to patch the Exchange cluster... I wouldn't hold my breath. If there is no 3rd party mail backup on that platform, that mail is gone.

Everyone on that mess should have been off a decade ago anyway.

The fact that they claim they have "restored access for thousands of users" leads me to believe they are somehow migrating data from the breached servers onto new ones?

I don't know. For my particular case (which I'll know better tomorrow morning) I think the damage will be pretty minimal. Those losing 2 days worth of emails is still pretty bad, not as bad as 2 days + however long the local desktop client wasn't keeping pulled down locally.

Fingers crossed that slider was all the way to the right (give me all the mail, baby!)

 

[Sky-Knight]

@brandonkick Beware that slider, because Outlook dies at 50gb of mail.

 

[YeOldeStonecat]

Official announcement of "Ransomware"

Rackspace says ransomware attack caused outage, warns of revenue loss in $30 million business line

The San Antonio-area company said its internal security team is working with an outside cyber defense firm to investigate.

www.houstonchronicle.com

 

[brandonkick]

Official announcement of "Ransomware"

Rackspace says ransomware attack caused outage, warns of revenue loss in $30 million business line

The San Antonio-area company said its internal security team is working with an outside cyber defense firm to investigate.

www.houstonchronicle.com

PLEASE at least let me get a backup of as much of my two hosted exchange boxes as I can get.... PLEASE!!!!!

I don't want to see them go out of business, but damn. I NEED THAT MAIL.


I'm sure thousands of other IT folk feel just the same way I do right now though....

 

[Sky-Knight]

I feel a little for those being well paid to get new systems online that should have been online a decade ago.

But I feel nothing for those stupid enough to utilize this service... at all. I'm one of the last properly trained Exchange admins left. VERY FEW are qualified to work on that mess, even less have the dedication to properly patch, service and maintain the things. Especially when the service rollups often claim to have completed while not actually modifying the code stored in IIS.

In short, if you don't work for Microsoft your odds of being able to keep the crap out of your Exchange server are slim to none... use M365 and hire a lawyer. It's cheaper that way anyway.

 

[brandonkick]

I feel a little for those being well paid to get new systems online that should have been online a decade ago.

But I feel nothing for those stupid enough to utilize this service... at all. I'm one of the last properly trained Exchange admins left. VERY FEW are qualified to work on that mess, even less have the dedication to properly patch, service and maintain the things. Especially when the service rollups often claim to have completed while not actually modifying the code stored in IIS.

In short, if you don't work for Microsoft your odds of being able to keep the crap out of your Exchange server are slim to none... use M365 and hire a lawyer. It's cheaper that way anyway.

Stupid is a bit of a strong word.

For a company really only needing "good" mail service, Rackspace offering exchange in a platform as a service fashion was quite nice.

Reasonably priced, and for the longest time it worked really really well. No major issues... until this one. It was light years ahead of the consumer based hotmail, outlook, yahoo mail, msn mail, aol mail.... whatever. Leaps and bounds better.


In todays world though, there is no reason NOT to just go straight to M365. For the same price (maybe even cheaper, I think) that what Rackspace charges... you get the entire 365 platform. All your office apps, teams, sharepoint, onedrive, and exchange mailboxes. For less money? Sign me up. I think we are or were paying like $13 or $15 a month per Rackspace exchange mailbox. We are getting M365 biz standard for something like $11 a seat.

 

[Sky-Knight]

Stupid is a bit of a strong word.

For a company really only needing "good" mail service, Rackspace offering exchange in a platform as a service fashion was quite nice.

Reasonably priced, and for the longest time it worked really really well. No major issues... until this one. It was light years ahead of the consumer based hotmail, outlook, yahoo mail, msn mail, aol mail.... whatever. Leaps and bounds better.


In todays world though, there is no reason NOT to just go straight to M365. For the same price (maybe even cheaper, I think) that what Rackspace charges... you get the entire 365 platform. All your office apps, teams, sharepoint, onedrive, and exchange mailboxes. For less money? Sign me up. I think we are or were paying like $13 or $15 a month per Rackspace exchange mailbox. We are getting M365 biz standard for something like $11 a seat.

That's precisely my point. 10 years ago, one could be forgiven for skipping M365 and going with Rackspace because they were coming off IMAP/POP3 garbage and it was the next logical step. M365 services after all hadn't quite turned a year old yet, being launched in June of 2011.

However, after M365 proved itself around the 2015 mark, it was time to GTFO because there was no value prop in staying, only increased liability. When the pandemic hit, that was the final warning bell. Anyone still on Rackspace at this point in 2022 is objectively stupid and not investing in their IT properly and as such will be paying out the arse to get back online once again because they failed to get the heck out of the way.

The IT companies helping these idiot businesses aren't stupid, they're smart and making money while they can.
The companies that no doubt ignored their IT company's recommendation to update their junk, they are very much stupid.

 

[YeOldeStonecat]

Personally I think Microsoft proved 365 within 4-5 years of 365 hitting the market. Let's see...I think it came to market in 2006...and we got invited to a sneak preview of the partner program (granted..stinky margins back then, before CSP came out). But...by 2010 we had quite a few clients moved from SBS to O365..mostly E3 then.

Going back to RackSpace, before 2010, it was still viable, although, by 2007, if someone needed to move from on prem Exchange, or get real email and grow up from POP/IMAP, we brought them to 365.

Before 2006, we had a few clients on RackSpace's hosted Exchange. And they DID have a cool "hybrid bundle"...which was rare. Have the big wigs on hosted Exchange, and have the general troops on lowly (aka cheap) IMAP mailboxes.

While RackSpace used to be a big grand daddy of hosted services....web, mail, DNS, virtual servers, and good old Jungle Disk, there was one company that really made the name of hosted Exchange prior to them, Intermedia!

But back to RackSpace, I'm guessing they didn't have Datto Siris for backup, else they'd have been back up by dinnertime last Thursday!

 

[brandonkick]

Personally I think Microsoft proved 365 within 4-5 years of 365 hitting the market. Let's see...I think it came to market in 2006...and we got invited to a sneak preview of the partner program (granted..stinky margins back then, before CSP came out). But...by 2010 we had quite a few clients moved from SBS to O365..mostly E3 then.

Going back to RackSpace, before 2010, it was still viable, although, by 2007, if someone needed to move from on prem Exchange, or get real email and grow up from POP/IMAP, we brought them to 365.

Before 2006, we had a few clients on RackSpace's hosted Exchange. And they DID have a cool "hybrid bundle"...which was rare. Have the big wigs on hosted Exchange, and have the general troops on lowly (aka cheap) IMAP mailboxes.

While RackSpace used to be a big grand daddy of hosted services....web, mail, DNS, virtual servers, and good old Jungle Disk, there was one company that really made the name of hosted Exchange prior to them, Intermedia!

But back to RackSpace, I'm guessing they didn't have Datto Siris for backup, else they'd have been back up by dinnertime last Thursday!

Out of all this, that's what REALLY puzzles me.

I understand getting compromised, but the fact that this hasn't been resolved by now tells us their business continuity plan was seriously lacking and obviously not well tested.

This is the main argument I made back in 2015 or 2016 when I got the local company I do SMB for to sign on with Rackspace (and get away from hotmail, gmail, ect...) and get a Synology NAS to host on prem data as "some kind" of upgrade to the "server" just being a workstation with a hard drive shared with everyone that got backed up if and when someone remembered to copy the whole disk to an external USB hard drive. That's the kind of crap that causes a business to close it's doors. Now they get to see it unfold with Rackspace. The humor in that is... Rackspace is a company I recommended to them to avoid the very situation Rackspace is in.

 

[Sky-Knight]

@YeOldeStonecat Memory sucks... use google: https://www.google.com/search?client=firefox-b-1-d&q=O365+release+date

O365 launched June 28th, 2011. The public beta release was in 2010. So while I have no doubt your progression through the products is as you recall, the dates are not correct. The platform is far younger than any of us buried in it want to think.

Office 2013 was the first release of Office to be "cloud influenced". And you have to be several updates in before it's cloud native, and a solid argument could be made that you had to wait until Office 2016 to be cloud native.

@brandonkick I swear half the reason we have M365 today is because Microsoft got tired of taking the blame for people being piss poor administrators. Now we have people administrating managed tenants that haven't a clue how Exchange works, much less the other components. That's both a good thing, and a bad thing. But we all also have so many other things to keep up with, I for one am glad for the extra time! As for rackspace... no corporation will invest in a product that isn't growing. It's not sane business to compete with Microsoft, they know they'll lose. So they just didn't invest in the platform and let it atrophy instead of actively moving people off like Godaddy is currently.

 

[YeOldeStonecat]

@YeOldeStonecat Memory sucks... use google: https://www.google.com/search?client=firefox-b-1-d&q=O365+release+date

Gah...I guess so....lol. Running on a couple-o-hours sleep so far over the past 2 days...