Hope you don't mind me posting this here Dave, but your busy in Pittsburgh anyway. ; )
Anyone receive any further word from Paul? Also does anyone know where the Redmen.com Facebook group went?
I got another update from Paul on Wed, 4/13:
Hello Again,
I won't detail all of the events of the past few days but it has been nightmarish. (the worst part dealing with all the plastic, corporate CYA B.S.) But I wanted to update you and I want you to know that I appreciate all of the encouragement and well-wishing, prayers sent and offers of assistance that I've received the past couple of days. I have not responded directly because of time constraints and focus.
Here is the status as of this morning Wed 4-13
On the positive side, I've managed to get all the services restored in another data center and DNS is (as far as I can tell) fully propagated.
Anything positive from the past couple of days is purely on the good deeds of some friends and surprising business associates that stepped up unexpectedly. A friend of mine who works for Dell in Texas went WAY above and beyond in helping to secure the equipment from the (former) data center. He voluntarily took charge and without any hemming or hawing including when Softlayer dragged him into their nonsense sending him on a wild goose chase and not letting him access the equipment directly. They insisted that I had to PURCHASE it from them and at best they would then ship it out to me (when? who knows). Having him onsite at least we were able to get one of their VPs on the phone and he approved handing off the equipment physically and he was able to find the last Fedex drop time (right near the airport) to get the data drives overnighted to me. Not a single complaint or bad word for the situation from him. Just a make it happen attitude which is exactly what you need and the type of person that you need in such a situation.
On Monday when it became clear that SoftLayer (our former data center provider) had zero interest in resolving the situation and was 100% focused on their own CYA, I contacted a local company that I receive other services from: Sonic.net Unfortunately normal channels were not prepared to offer me the type of instant solution necessary. I shot off an email to my account rep who I had no expectation to hear from until the following day since it was already after hours. Surprisingly I got a quick response. He then proceeded to get me on the phone and conference in their overnight Data Center shift leader and the two of them made sure that I had equipment online and live to the internet complete with a block of IP addresses. Literally within 15-30 minutes. This enabled me to work overnight with my admin team to get all of the DNS, key domains, email and services restored.
On the other side of things, Softlayer is still holding me hostage. The CYA dance ensured that there was no other option for me short of a legal one. And no offense to my attorney friends reading this, but once you get the attorneys involved things leave any "let's solve this" scenario and certainly not a "lets solve this in a timely fashion." Yesterday the drives got sent to me and the next step is attempting data recovery. I am hoping that they actually gave me the real drives and didn't further compound things by either sending me random blank drives as part of their ongoing CYA, or do anything further to wipe the drives in an attempt to cover their tracks and not present me any evidence of their blunder.
I am now waiting for Fedex. The tracking says it is on the truck for delivery. I should know pretty quick if A) they sent me either completely blank or drives that they intentionally wiped B) any quick data recovery is possible. "B" would allow us to get back online very quickly. I won't touch "A" right now. The third scenario would mean a more extensive data recovery that could take days to weeks and no guarantees that it is any better than A. In some ways worse since it would require more expense and time wasted and the ongoing uncertainty.
Sot that's it for now.
Paul Massell