Wednesday, December 7, 2011

Blog feed for more than 500 posts

The folks at Blogger, bless their hearts, erect several obstacles to creative use of your blog's feed.

Some of us use our feeds--our whole feeds, not just the most recent stuff--to make an index of our blogs, or to reverse the order so that it begins with the oldest post.

The default Blogger feeds, however, show only the most recent 25 posts. This is pretty easy to get around: you just append "max-results=999" to the feed url, which overrides the 25-post limit (replacing it with a a larger one). So the feed for this blog, breaking the 25-post barrier, is
http://too-clever-by-half.blogspot.com/feeds/posts/default?max-results=999
It turns out that Blogger has a second, more-serious limit of 500 posts per feed. Here's how that works, and how to work around it if your blog has more than 500 posts.

This limit has probably been in place for a while. I missed it because it is not documented and also because I do not have more than 500 posts on any of my blogs. (Yet.)

Fortunately, this limit is also flexible, in a way. All of your posts are in your feed. However, you can only see them in 500-post increments.

For instance, if the first 500 posts of your blog feed are here:
http://YOURBLOG.blogspot.com/feeds/posts/default?max-results=999
the second 500 posts are here:
http://YOURBLOG.blogspot.com/feeds/posts/default?max-results=999&start-index=501
The "start-index=501" parameter (after the ampersand, which you must add) says to start the 500 count at post no. 501. If you have 809 posts in your blog, this will yield the oldest 309 of them.

Note the "max-results=999" parameter is still needed to override the other default limit of 25 posts.

This is a lot like the recent decision of Yahoo Pipes to paginate its feeds, and the remedy is similar.

You must treat each 500-post increment as a separate feed, manipulate each as you see fit, and then paste the scripts for them serially into your blog to create the entire series. (Use a service like feed2js to create the script for you.) You can also merge the feeds at Yahoo Pipes, if that works for you.

By the way, there is a similar 500-post segmentation of any label feed, and a 200-comment segmentation of the comment feed. In both cases the "start-index=" parameter gives you access to your whole feed, in chunks.

This is a nuisance and if it pushes you to say its not worth the trouble I do not blame you. However, with this workaround it is possible to get at your whole feed, if you need it, Blogger's limits notwithstanding.

Blog feed for more than 500 posts

The folks at Blogger, bless their hearts, erect several obstacles to creative use of your blog's feed.

Some of us use our feeds--our whole feeds, not just the most recent stuff--to make an index of our blogs, or to reverse the order so that it begins with the oldest post.

The default Blogger feeds, however, show only the most recent 25 posts. This is pretty easy to get around: you just append "max-results=999" to the feed url, which overrides the 25-post limit (replacing it with a a larger one). So the feed for this blog, breaking the 25-post barrier, is
http://too-clever-by-half.blogspot.com/feeds/posts/default?max-results=999
It turns out that Blogger has a second, more-serious limit of 500 posts per feed. Here's how that works, and how to work around it if your blog has more than 500 posts.

Saturday, December 3, 2011

Feed "pagination" hobbles Yahoo Pipes

The quirky but useful Pipes web service has added a new kink, with implications for bloggers that use it to reverse the order of their posts.

Running your blog feed though the pipe I built will flip the order, and until recently you could then just port the pipe's feed to feed2js to show your blog in the order you want.

Frustratingly, Yahoo Pipes has started to paginate its feeds in blocks of 100 posts. So the feed url you get from Pipes only displays the first 100 posts in the series. To see the next 100 posts, you have to add the following to the feed url:
&page=2
You then run both feed urls, separately, through feed2js, and mash the results together so they look like a continuous feed.

Pain in the neck, but it's actually worse than that. You have to do that for each block of 100 items in your feed, so if you have 600 posts on your blog that means six separate feed urls, up to &page=6.

Heck, it is worse than that. If you keep blogging (and I hope you do), you must periodically add 100-post segments as you grow. You can't just set it up and leave it to do its magic as you could previously. There is potentially no end to the process.

Bad Yahoo!

Before I walk you through how to do this (and it's not so bad, its just that it should not be set up this way at all), lets review some other options.

First of all, you might want to paginate your own blog journey by year, as I do on my other blog. A script with hundreds of blog posts can take a long time to load and breaking it up makes it more manageable.

If you are going to do that anyway, and if you do not have more than 100 posts in any year, then you don't need to worry about this new curve ball from Yahoo at all. Plop your feed into my pipe, segment the feed by date for each year, and go. If there are less than 100 posts, the issue does not arise.

Second, Yahoo does not paginate its JSON object--it give you the entire feed in that format. If you have facility with javascript you can whip up your own script using the JSON. Unfortunately, feed2js does not work with that format.

I think the JSON-javascript option may turn out to be the best solution. But so far my Blog Journey hack, for all its faults, has not required anyone to be a programmer. (Granted there is a script, generated for you by feed2js, but all you need to do is copy and paste it into your blog. Feed2js does all the heavy lifting and that, along with its great user interface, is why we love it.)

One great accomplishment of all these cloud-based web services, such as Pipes and Blogger itself, is that they put a friendly user interface between you and the underlying code. I try to honor and preserve that. I feel I am writing for some pretty smart people here, but I draw the line at actually programming.

So here's how to work around Yahoo's unfortunate pagination problem.
  1. Run the feed through my pipe and get the feed of the pipe output, as described here.
  2. Using a service such as feed2js, turn that into a script and paste it into into your blog.
  3. For each additional set of 100 posts in your feed, add &page=n to the feed url, where n is the page containing the set (starting with 2 for posts nos. 101–200).
  4. Run that though feed2js and add the code to the script you've already pasted into your blog in way outlined below.
  5. Repeat that for all the pages in your feed.

If you just paste the scripts one after the other, there will be a space between them in the output they generate on your blog. To avoid this, eliminate the spaces after the <script>...</script> and <noscript>...</noscript> tags. (Lots of stuff between those tags; don't mess with it.)

Those spaces make it a lot easier for the human eye to parse, but the script does not require them to run.

This whole episode is another reminder, not that we needed one, of the drawbacks of using third-party web services.

I still think that doing so avoids analogous drawbacks of trying to hack Blogger with scripts, but this sure has not been a good year for the third-party approach.

Feed "pagination" hobbles Yahoo Pipes

The quirky but useful Pipes web service has added a new kink, with implications for bloggers that use it to reverse the order of their posts.

Running your blog feed though the pipe I built will flip the order, and until recently you could then just port the pipe's feed to feed2js to show your blog in the order you want.

Frustratingly, Yahoo Pipes has started to paginate its feeds in blocks of 100 posts. So the feed url you get from Pipes only displays the first 100 posts in the series. To see the next 100 posts, you have to add the following to the feed url:
&page=2
You then run both feed urls, separately, through feed2js, and mash the results together so they look like a continuous feed.

Feed-based hacks get a new lease on life

Alan Levine, originator and keeper of the useful but endangered feed2js.org web service, announced last week that thanks to

the generosity of people who have donated financial support, and one anonymous donor in particular, I have sufficient funds to keep Feed2JS running at least through June 2013, and maybe longer.

Feed2js turns rss feeds into scripts that will run on blogger.

If you manipulate your blog feed to, say, reverse the order of posts, you can then run it through feed2js and just paste the result onto a static blog page or sidebar gadget. No javascript knowledge required. Genius!

Alan announced last summer that the popularity of the service had made it too expensive for him to fund out of pocket.

His latest news of a year-and-a-half reprieve (in dog years, that is more than a decade) is welcome news.

The full story is on Alan's blog.

Feed-based hacks get a new lease on life

Alan Levine, originator and keeper of the useful but endangered feed2js.org web service, announced last week that thanks to

the generosity of people who have donated financial support, and one anonymous donor in particular, I have sufficient funds to keep Feed2JS running at least through June 2013, and maybe longer.

Feed2js turns rss feeds into scripts that will run on blogger.