Blog

Digest serving speed problem resolved

Users will have noticed that digests were being served rather sluggishly over the past few days. This situation has annoyed us, so we've been straight on the case to sort it out. The problem arose because our 'application server' deals with a lot of things at once. While 90% of requests are served from our very fast memory cache, the remaining 10% can 'sticky' up the system. This means the 90% of requests slow down a lot.

Today we've changed how it works. Now we've made it so that EVERYTHING is cached, even if the cache is out of date. We'll serve whatever is in the cache, and add that digest's ID to a special list of digests to refresh 'behind the scenes'. The cache is then updated directly, rather than hogging up the user's time. The slight disadvantage of this is that digests may seem to update a little slower than before (although not by much), but the advantage is that they can be served a lot faster than ever before.

We will be giving paid subscribers a way to get the best of both worlds shortly :) For now, however, paid subscribers still get significantly more regular updates and are always at the 'top of the pile' on our crawlers and cache updaters.