Great stuff guys and I am more than willing to help Paul out as well...Unfortunately, I have a lengthy update today and Paul really got screwed:
I've received a lot of inquiries and offers of assistance. Thank you.
I stopped doing the daily update when it was clear that this was not going to be resolved quickly and frankly I have barely moved from my computer for over two weeks and slept very little. On my birthday I spent a good part of that being castigated by a customer - including getting screamed at over the phone for about an hour! So I have not had a lot of time to be updating or responding to people. I apologize for that.
The three issues here that are affecting the time frame are: Technical, Financial and Legal.
I've hired a couple of independent third party companies in this process and have gotten an attorney involved. The current status is this:
1) Let me start by saying that I have been working with, hiring, administrating, owning, data center/networks and specifically those related to web/data hosting, for 20 years. Even in the days when a microsoft windows server would need to be rebooted several times a day as a standard, I have never witnessed or in any way been a part of something as extreme as this.
(in my opinion) the original data center made a huge technical blunder but worse, took this to ridiculous extremes by attempting to cover it up and deny responsibility. When after two days they had no reasonable answer (and remember that in this business it is 24/7 and these data centers brag and market about "five nines" i.e. 99.999% up time) so not even being able to talk to the guy in charge until late into the second day of complete outage is completely unacceptable. Then his answer to me "I have to investigate." Shouldn't the investigation have begun immediately not two days later? So I won't rehash this further here but on the technical side, I have hired a leading data recovery company. And completely independently, they confirmed my conclusion as to what had occurred - what the data center had denied or more specifically "I see no evidence of that." (sounds like an attorney wrote that line for him)
2) This is quite a costly situation. The data center would not let a Dell technician / employee; access the hardware that my data was on even though it was Dell equipment and while I don't know this for a fact, this data center claims to run about 60,000 servers in Dallas where my equipment was housed, and I'd venture a guess that a large percentage of them are Dell - and further take a wild guess that Dell techs are in the data center on a regular if not daily basis... They gave me a line about not allowing anyone in the data center - this after providing him/me with the wrong address then him (the Dell tech) going to the correct address despite the wild goose chase.
Ultimately their solution was that they would sell me the equipment at several times Retail - that is "NEW" retail. Example: a single hard drive not even produced any more and certainly not new, but last competitively priced around $90 new, they priced at $500. They said (with a concerned sounding voice - and they had a young woman present this to me) they'd do me a favor and sell it to me for $250. Really swell of them! I had to get a vice president of their company to get approval to get that down to $175 per drive and the impression that I got was that by doing this, I should consider them on par with Doctors Without Borders and Mother Teresa. He also apparently had to get approval from someone (legal maybe?) to do me this enormous favor - no matter that the clock was ticking and that 99.999% was long gone so had to hang up on me to call me back even when I told him that I was willing to stay on hold for as long as it would take. In my opinion this entire scenario was part of the orchestrated effort to build their plausible deniability and some ridiculous attempt at making themselves look good regardless of whether or not the end result was further sticking it to me. (as I said ridiculous)
3) With an independent third party, that is an industry leader in data recovery, essentially confirming exactly what I suspected occurred, I believe I have a rock solid case to compel them to cover the expenses incurred and lost. Just data recovery is approximately $10,000. But the third of my list: Legal, means that even the best outcome has no specific timeframe and of course the attorneys need to get their cut, so my only option is to pay out of pocket until which time the legal route brings a resolution. And of course there is no guarantee there.
And after the initial deposit and report the data recovery company states that they believe they have a "high level of confidence" that 3/4 of the data can be recovered. But I have no idea which 3/4 that will be. If that 75% of the total includes the operating systems and various software that runs the servers and/or older backups or deprecated files/data - that could mean that the actual important data could be recovered to a much lower percentage - even 0%. The other thing here is that there is ZERO guarantee that ANY data will be recovered. Only a "high level of confidence" with a big - no guarantee asterisk
So into the third painful week of this, I am waiting for the data recovery company to give me the results of their efforts to recover the data. Once that is complete and they are paid, they will provide the restored data. Once I receive it, I can get an idea of what is involved and how long it will take to begin restoring the data into its proper place. Needless to say, I am not restoring this into the same equipment and data center but I have the infrastructure completely ready/live and in place to accept the new data. I'm also using the services of three distinct and separate companies to house, monitor and backup everything so should some sort of perfect storm scenario occur again, we'll have options for a quicker resolution. I'm also going to be using Amazon web services moving forward (4th service provider) to both improve network performance as well as keep segments of the data/files diversified.
I am sorry I don't have anything more positive or specific for you. I appreciate the generosity of various sorts and the offers of financial assistance. Let's hold off on that please. I'd be embarrassed to accept money from anyone while everything has been offline for over two weeks and I don't even have a reasonable eta for restoring or replacing things. I am however, 100% committed to that (restoring or replacing) in one form or another.
Regards,
Paul