The long version: Saturday afternoon, I decided to upgrade one of my development servers from 7.04 to 8.5.1 today. I am pleased to say that as far as I can tell the upgrade was easy and fast. Once I backed up my data, it only took me about 10 minutes to install 8.5.1 and let the server do its thing at startup. I then restarted the Domino service and all appeared fine. How's that for a seamless upgrade? Sweet!
The last step was to reboot the server to confirm that the Domino service would restart automatically. As I went to do this, I noticed the pop-up from Microsoft, telling me I had Windows updates pending. I decided to go ahead and apply these and reboot.
That's when the fun began.
Apparently the server boots part way and hangs, saying that it is at step 3 of 10 for the Windows update.This dev server is a dedicated server at a major hosting provider. Unlike some of the fine hosting service providers, my provider just rents boxes and it's up to you to manage your own box. Their support is pretty much limited to a power cycle and, if all else fails, a reprovision (complete wipe of the OS and your data and reload of the box). While they make the reprovision process as easy as one click, I did not want to have to start over and lose a current 30GB backup. So, I have been working with tech support. It looks like I may need to reprovision the server and start over. Ouch!
The good news in all of this is that I have Lotus Notes/Domino.
Thanks to Notes and Domino, things continue to run. Mail routing and replication has failed over to the stand-by server and I am the only one a little stressed. If I do need to reprovision the server and start over, I will restore an old backup (any will do) and wait for the data to replicate across from the live server. This means that I could be running within 30 min of reprovision and have my data back in 3-4 hours. Try that with any other system.
I cannot even begin to imagine what this would have been like if this were another brand of collaboration server that had crashed. In any case, it wasn't the collaboration server that crashed, it was Windows Server.
1. The Domino 8.5.1 upgrade process was easy.
2. Lotus Notes/Domino can't be beat for redundancy, failover, and ease of recovery
3. From now on, I will take the time to FTP data to another server before I do an upgrade and not temporary backups on the same server -- even for a 10-minute upgrade. A Windows update could wipe out your OS and you could lose everything.
4. I will explore another OS for Domino. I know Windows well, I'm not convinced that an update hiccup couldn't take down a Linux box in the same way.
5. I will think about either moving this in-house or to an ISP that is in the business of hosting Domino who can take care of something like this for me.
Update 01:55 AM PST: The tech just called me back. Server OS is hosed. They cannot even login. They offered to pull the drives and move them to another server. I requested a reprovision instead - I'd rather start with a clean box. When I wake up, I'll FTP my backup and install directory to the server and reload Domino. The good news is that I don't need to worry about my data. I should not have lost anything. Thanks to Domino and Replication, I'll be back up and running with no work on my part. I'll just need to wait for replication to finish.
Update 11:30 PM PST. Once the ISP reprovisioned the server, it took me just over 30 minutes to download and endless stream of Windows updates and tweak the settings the way I like them. It took 3 hours restore an old backup to the server via FTP, another 10 minutes to download Domino 8.5.1 and 10 minutes to deploy Domino 8.5.1 which was the easiest part of my weekend. With Domino up and running (using a week old back-up, mind you) I watched as the servers replicated and all of my data was restored without my involvement. Now that's fantastic. So, all in all, my experience is that the Domino upgrade itself was painless and took me about 10 minutes.
Thank you Lotus.