From:	Clement Mariaselvanayagam
Sent:	Tuesday, March 27, 2007 8:57 AM
To:	Terri L. Famer
Cc:	Yvonne Engel
Subject:	Summary on Disaster week of March 11th
Sunday, March 11 '07 

MH called me from the Foundation and indicated that he could not get Email and access the Bradley files. 
I came in and noted that the \\BRADLEY server was down ... with a black screen error reading ...

"<Windows 2000 root>\system32\ntoskrnl.exe missing or could not be found"

I rebooted the machine and received the message that

"SCSI ID 1 had failed", it is back up and operational, press F9 to rebuilt

Hard drive was functional however the rebuilding process of the redundant RAID 5 config would error out.
I tried to get into recover mode by running the RAID ARRAY CONFIG TOOL through SMARTSTART, 
however the recovery failed half way through the process.

I tried to re-install and repair Windows 2000 server software by installing from the CD, but could not proceed into the setup section.
Tried to start server in Directory Restore Mode which also failed with the same server errors.
Tried the Last Known Good Configuration option and it failed with the same error.
I let the RAID 5 Config tool run and went home, hoping that if the rebuilding process completes, 
I may have a chance at resurrecting the server.  
The rebuilding process should take between 30 mins and 6 hours depending on the amount of data on the hard drives.

Monday, March 12 '07

12:30am to 1:30am

Came in too see if the Hard disk had rebuilt, it did not, it had failed half way through.  
I decide to let in run for the full 6 hours and I went home.  I rebooted the machine and continued to receive errors on SCSI ID 1.  
I suspected the Hard drive had failed and wanted to call Hewlett Packard on Monday and have a replacement sent to us ASAP.

Monday, March 12 '07

Came in at 6 am, RAID rebuilt did not work.
Called HP, reseated hard disks, took memory out and reseated them as well.  I rebooted and RAID was still not rebuilding.  
HP suggested that I upgrade the BIOS.  So, I downloaded to the latest BIOS upgrades and also downloaded and created SMARTSTART CDs.  
I indicated that I have been getting hard drive errors.  I suggested the Hard disks had hard faults.

I asked YE to order 1 HD + spare from Masterlink

Tried to run WIN2K setup to bring Recovery Console again but still could not, went into system utilities and sent the log files to HP

HP called me back and based on the log files there were some hard fault errors with HDisk 1 and 2.  
This explains why I was having a hard time rebuilding the RAID 5, since 2 hard drives had failed and 
this meant that the redundancy had failed and we could not rebuilt the hard drive to original state.

Network no longer has DNS and DHCP roles, so client machines could not get on the Internet.  
I transferred all 5 FSMO roles to the \\RADER server.  Also setup \\RADER to be the new DHCP and DNS server, 
so people could log in and finally get on the Internet.

Paul Benedict of Masterlink came in and ran the Memtest86...memory passed…
he was concerned about whether Motherboard or SCSI Raid Controller card or HDisk could be bad.

We make the collective decision to try and redo the Server, since we could not even get into setup to try and salvage the system, 
and time is of essence.

After Paul leaves, I run SMARTSTART, and run System Utilities and wiped the system clean.  
Setup finally runs!!!
I install Windows 2000 Server +SP4 with brand new larger partitions.  I kept it in a different Workgroup, 
but gave it the same Computer name to make sure that we could install Exchange server 2000.  

Prior to joining the domain, I reset the computer account, and transferred and seized the FSMO roles.

I joined \\BRADLEY to the domain and promoted it as a domain controller

Tuesday, March 13 '07

Installed Exchange server in Disaster Recovery Mode, this process needs the PRIV and PUB .edb and .stm files, 
which were not in the backups.  
This means that our exchange recovery will have to take on a different approach.  
Server's hard drives times out and cause freezes and is very slow.
Uninstall of exchange fails…
Decided to enlist help of exchange server expert and AD registry editor - David Michalski

The hard drives arrived!!

I replaced both hard drives on SCSI ID #1 and #2
Re-installed Compaq SMARTSTART, configured RAID 5, re-created the logical drives to Match the old server
(This should help w/ exchange restore procedure)
Installed WIN2k + SP4
Installed Outlook Client
Needed to restore Active Directory from backups and restore FSMO roles on the server.  It has to have the same server name.  
However this can conflict with Active Directory due to new security identifier (SID).   
Their roles need to be switched and server needs to be promoted and AD needs to be hacked.
Hence, AD Hacker expert David Michalski comes through, who is fearless about using the ADSIUtil tool.
Before we could do that we had for restore the system states which was unreadable from the tape backup, 
due to some security issue with our BackupExec Program.  We had to give it Local administrator privileges and were eventually able to read the tapes - 
we restored the AD system state and proceeded to Install Exchange.
However, we recognized that .edb and .stm files of Exchange server were not getting backed up, along w/ missing Public Folders.  
Mailboxes were only backed up MAPI client

David said he would talk to Jim, the Exchange Server expert and come back with ideas
I stayed till 3:30 AM and restored H: and U: drives.  I put them on the Bradley server and reset the DHCP and DNS settings.
We don’t have email but at least we have our Network back.

Wednesday, March 14 '07 - Thursday, March 15 '07

Uninstalled Exchange from the Disaster Recovery Mode following  MS TECHNET document. Moved all of U: drive which is now mapped to the //RADER server.  
Reconfigured the login script and fixed network programs to point to the new server
David suggested we install the Exchange on a new install mode and by using the MAPI client we can bring individual mailboxes in.
However, this method failed and we eventually deleted each mailbox, recreated them, and sent an email to activate the mailbox.
Had to import mailboxes one by one due to MAPI backup method, as I imported each mailboxes, I went to each workstation and set it up so it would be trivial to the user. 

I setup outlook, and recreated shortcuts and address book locations

Thursday, March 15 '07 - Friday, March 16 '07 

Blackberry was not working
Apparently it had AD attributes in Grebe, Lempke, and Schmidt mailboxes
I am not experienced in editing registries on the server, so I enlisted that help of David Michalski who came to help.  
We had to delete the mailboxes and recreate them for Blackberry to work with Exchange agent

Re-installing Blackberry did not help…
Removed attributes and recreating the server helped
Had to recreate the SQL MSDE database, which was corrupted, and we had to clean the database

Reset Blackberry devices and added them individually to the Server.
Restored the Bradley calendar, and contacts of VP from Dan's Blackberry
Restored Bradley Calendar shortcuts in everyone's workstations