Blog
New template tag – COUNTER
Want to number each item in your digest? A few people have asked for this over the past few weeks, so we've added a tag to do it. Just edit your digest's template and include the %COUNTER% tag. It'll start from 1 and work up to however many posts are in the digest. This isn't useful for everyone, but if you need this feature, it's now available :)
If you want to suggest any other tags we should implement, please do so here.
Big performance increases
Some major changes have taken place 'behind the scenes' this evening. Our crawling system has moved to a 'queue' system for adding new items to our database. This means our database is now a lot faster, and we can crawl several times quicker than before. We implemented the 'priority' crawling for paying subscribers mainly because the volume of feeds used by free users was too much for our poor crawlers to deal with.. No more! For the past few hours, as soon as feeds have hit the queue, they've been processed successfully. Paying subscribers do still get to the top of the pile, but all users should certainly notice some major improvements in crawl times. These changes may have affected certain items in certain digests, particularly on items crawled for a period of a couple of hours today. If there are any major problems with your digests, please post in the forum or leave a comment here.
Coupled with the performance increase is a new feature. Due to the new queueing system, we now know precisely when a digest can be recached, as we know when its constituent feeds have been updated, to the second. This means your digests will now update instantly, as soon as the feeds for your digest have been crawled. Say goodbye to hitting refresh multiple times!
Furthermore, these improvements have been made in anticipation of the FeedDigest relaunch, which will feature a whole ton of goodies for you.. not least of all a 'guaranteed crawl times' service for paying subscribers. You'll be able to tell us at what intervals you want certain feeds to be updated, and we'll do it regardless of if anyone is accessing your digest. All in a day's work.. :)
Paying users now get preferential crawling
We've just launched an improvement to our feed crawling system where feeds used by digests belonging to higher level users are given a significantly higher priority. This means the regularity of updates should improve significantly for paying users.
The amount of 'extra priority' goes up by account level, although since the vast majority of Feed Digest users are free users, simply having the entry level Basic account will leapfrog your feeds to a high position in our crawler. If you want to sign up, plans start at just $11.99 per year, click on the "Upgrade" link in the right hand side of your control panel.
Upgrades re-enabled; upgrade today!
You can now upgrade your account automatically once more. Simply go to your control panel, and on the right hand side you'll see an Upgrade link. Use this if you want to be able to use FeedDigest without linking back to our site, if you simply want to support us, or if you want the extra features our higher account levels can give you.
As a bonus, all users who pay for an account in January will receive some nice benefits when the new version of FeedDigest re-launches (with new, extra premium pay-for features -- but don't worry, you free users won't lose anything you already have :)).
Major performance upgrades go live
It's all back-end stuff, but today we made some significant database changes to increase performance and capacity. This meant FeedDigest had to go off the air for two small periods today, so we apologize for the inconvenience. We try not to ever take it down, but the benefits from these changes were too big to put off.
We also had to stop our crawlers for about six hours, as we were more interested in digests being served reliably, than keeping the digests up to date, and we had to prioritize. The crawlers are now back up and running and busy running through the backlog. The database updates mean that crawls should be a lot faster, so we may (and there's no promises yet!) let you recrawl your feeds more than once every thirty minutes.. (indeed, we already have it running at every twenty minutes on the backend for 'test' purposes, but may lower this further..)
So, it's all go here, but the changes remain on the backend for now. Perhaps (finally?) we can roll out the more exciting features if this database upgrade truly gives us the stability we need to grow further and to ensure we can handle hundreds of new users day by day. Thanks for your support; we truly appreciate it.
Added: Peter Cooper has posted information on his blog about the changes made today.
Corrupted digests?
Some users may have experienced difficulties in updating digests this weekend. We've been making some minor architecture upgrades, as well as resolving some issues with our Web server, so if you've ended up with corrupted, unworking feeds, then do try rebuilding them as it should all work now. Thanks for your patience.
%TIMEAGO% tag added to templates
Thanks to a suggestion by Daniel of Southern Cross Ventures, FeedDigest now has a tag which will tell you how long ago an item's date was in the past (or on some broken feeds, when in the future!). So, for example, instead of simply listing the date with each item like "20th February, 2005".. if you use %TIMEAGO% in your template, you could get "67 days and 5 hours ago".
%TIMEAGO,FULL% gives you the most precision, e.g. "5 years, 2 days, 12 hours, 7 minutes, and 3 seconds ago"
%TIMEAGO% gives you middling precision, e.g. "5 years, 2 days ago"
%TIMEAGO,SHORT% gives you the least precision, e.g. "5 years ago" or "2 days ago" or "2 minutes ago"
Simply manually edit your digest's template from the control panel, and you can begin using this feature immediately.
Please note that %TIMEAGO% will not be up-to-the-minute accurate due to FeedDigest's caching. Depending on how often the items in your digest change, and the level of our caching at the particular time, times between updates will vary.
Mega fixes and mega apologies!
We have just experienced one hour of downtime. This was, sadly, unavoidable. Our primary database server reached melting point. Queries were taking a couple of seconds to run.. and that's fine at low loads, but when millions of requests are being made, it causes some big problems. We want queries to be running in a twentieth of a second, but it was running about 100 times slower than that.
We made some big changes, including moving the database to a separate SCSI drive, flushing most of our super old irrelevant feed items (no 'live' feed items were erased), and rebuilding all our indexes. We've also optimized the SQL powering most of our digests to the bare bones. We have also made some improvements to the memory setup.
When the server was started up again, it was serving rapidly and I was impressed with the increase. Then I noticed the cache wasn't running! It was serving every single FeedDigest request straight from the database and still had very good performance! I've fixed the cache again now, and the load is ridiculously low (for the techs, we were averaging 1.5->2.5 load - dual CPU - and now we're at 0.1!)
There are still changes to be made, but we'll try to make sure they don't infringe on your service too much. We just want to roll out cool products rather than have to focus on the infrastructure too much, so it's great things are now running smoothly.. just give things a few hours to fully sort themselves out on the feed crawling front now as our crawlers catch up.
Upgrades and updates
Better limits on free accounts. Free accounts now support mixing of up to five feeds at a time (previously three), and outputting of up to 20 items (previously 15).
Faster crawling of feeds. Today we've been working hammer and tongs to resolve one of our largest problems, and we've done it. We reached a point where the number of feeds we were having to update throughout the day became too much for our crawling server, and we needed to split the job out. Now we have four machines doing the work, and have totally rewritten the routines to be faster than before. Undoubtedly there'll still be some crinkles to work out, but it's processing feeds faster than ever.
ISO-8859-2 support. Several users have asked for ISO-8859-2 support, which means Feed Digest will output digests in a format used by many Eastern European languages. Unless you specifically asked for this, this feature might not interest you :)
Dutch language added. The Dutch language has been added to Feed Digest and day and month names can now appear in Dutch. Many thanks to PaulZ from the FeedDigest forums for requesting and helping implement this feature.
Better Atom 1.0 compliance. We might not be 100% yet, but our Atom output is now (mostly) validating as Atom 1.0.
Moving to nightly builds
We've begun to get a solid grasp on some of our old infrastructure issues and have been making lots of improvements to our "bottom end" over the past month. Now we want to begin focusing on the true application, the one that you see when you log in, once again. To do this we're making some minor adjustments so we can rapidly update FeedDigest whenever we want, without worrying about the effect of any changes or updates we make. So, soon we'll be moving to "nightly" builds, where as soon as we've checked a new feature, it's live for you. More news as this is rolled out..
(Update 20th October.. this is now live, so expect lots of rollouts soon.)