load balancing - Question regarding uptime strategy for website

I'm working on a website where we need to have a good deal of uptime. Specially in the short bursts (15-day periods) that the website's events happen.

The page is dead simple and can be served almost completely off an html cache. While there is a php based part, this is not mission critical and in case of failure, we can live with the cache for a say 20 minutes until the eventual problem is sorted out. More than 20 minutes won't really work as the site hosts a live score board among other things.

We had successful deployments in Amazon using multiple EC2 with elastic load balancing and also using the Rackspace cloud (Cloud Sites and Cloud Servers).

For this particular website we'd like to have the app running in these two providers at the same time but just serve the pages from off one of them. In case the main provider fails just move to the other and keep going.

I know how to make everything work in just one provider. What I'm not clear on how to achieve is the actual switch from one provider to the other. For example, if I CNAME myapp.com to a domain in Rackspace and it fails, when I change the CNAME to point to Amazon, a great deal of users will aready have their DNS cached to Rackspace and the whole thing would be pointless... this is one of the many questions that I have...

Any help is greatly appreciated... tips, advise, gotchas, anything is welcome...

Answer

I would think that either Amazon or Rackspace alone should give you the uptime that you require. The whole point of the cloud is that you already have a high availability situation. If your Amazon or Rackspace hardware fails your image is restarted on other hardware. Have you already had an uptime problem or are you trying to solve an issue that hasn't yet occured?

If you have massive surges of traffic, and a lot of static content I think you should consider a CDN. Edgecast has reasonable prices and an excellent network. All your static content could get served from their geo redundant servers and help you with site availability.