Blog

Downtime and the solution

Last night there was approximately ninety minutes of intermittent service from FeedDigest which caused a few problems for many users. The service was fully up and running again pretty quickly, but obviously there was an impact. I am still investigating the cause of the problem.

Lessons learned:

  • getting an e-mail-to-phone account so I can learn about downtime the second it occurs!
  • not to take a day off because that's always the one day things go wrong ;-)
  • get our new proxy system implemented ASAP

The new proxy system is an interesting one. From a year of RSS Digest, and now Feed Digest, I've learned that having 24/7 rock solid service delivery is of utmost importance, especially when you have paying customers.

At FeedDigest's level though, it's not viable to have rooms full of servers (not just yet anyway!) running the heavy-load FeedDigest software. It is certainly viable, however, to have multiple lighter systems in place to act as proxy/caches to FeedDigest. We'd simply associate app.feeddigest.com with a round robin DNS so that a random proxy/cache server is hit for each request, and then that does the talking to the main FeedDigest servers. In that way, the service never appears to be down, even if a backend server is! The 90 minutes of issues yesterday have underlined that this should be high on the priority list now. Thanks for your support, notifications, and help in troubleshooting.

Secondly, keep your eyes on this blog for a major announcement in the next few days. Yes, FeedDigest is about to hit the big time. It was all signed and sealed several days ago, and it's almost ready to go public..

3 Comments

  1. Thank you so much for this open and honest communication. While a microchip implanted in your brain so that we could all know what was going on at the instant it occurred might be even better, this kind of blog post and the damage control at 90 minutes is fantastic. I have stopped freaking out.

    Posted August 31, 2005 at 1:35 am | Permalink
  2. FeedDigest is about to hit the big time!?!?!? You certainly deserve it for the quality service you’ve provided.

    I’m all a-quiver with excitement! Do I detect a hint of major investment? Or maybe partnership?

    –Simon.

    Posted August 31, 2005 at 9:24 am | Permalink
  3. 90 minutes from the problem arising to a solution…kind of puts other, paid for, “BIG” web applications to shame. Well done.
    Eagerly awaiting the news you are teasing everyone with.

    Posted September 1, 2005 at 12:55 am | Permalink