Blog

Big performance increases

Some major changes have taken place 'behind the scenes' this evening. Our crawling system has moved to a 'queue' system for adding new items to our database. This means our database is now a lot faster, and we can crawl several times quicker than before. We implemented the 'priority' crawling for paying subscribers mainly because the volume of feeds used by free users was too much for our poor crawlers to deal with.. No more! For the past few hours, as soon as feeds have hit the queue, they've been processed successfully. Paying subscribers do still get to the top of the pile, but all users should certainly notice some major improvements in crawl times. These changes may have affected certain items in certain digests, particularly on items crawled for a period of a couple of hours today. If there are any major problems with your digests, please post in the forum or leave a comment here.

Coupled with the performance increase is a new feature. Due to the new queueing system, we now know precisely when a digest can be recached, as we know when its constituent feeds have been updated, to the second. This means your digests will now update instantly, as soon as the feeds for your digest have been crawled. Say goodbye to hitting refresh multiple times!

Furthermore, these improvements have been made in anticipation of the FeedDigest relaunch, which will feature a whole ton of goodies for you.. not least of all a 'guaranteed crawl times' service for paying subscribers. You'll be able to tell us at what intervals you want certain feeds to be updated, and we'll do it regardless of if anyone is accessing your digest. All in a day's work.. :)

More information (for technies/programmers, mostly) here.