Unicast

Thomas asked me to explain what Microsoft's Simple Sharing Extensions for RSS and OPML spec, or SSE, would be useful for. Here's an example of how SSE could help us build Photo Commons.

Here's how Photo Commons could have worked: For a given tag that we track, for example reboot7 on 23 or reboot7 on Flickr, we grab an RSS feed for that tag (23 and Flickr), read through the feed, and add any previously unknown photos to our database. If a photo was previously in our database, but not with that tag, we just tag our version.

This doesn't work because RSS feeds typically only include a handful of latest entries — 10 for the Flickr feed, 20 for the 23 feed. Of course feed providers could provide more entries in their RSS feeds, but this would make downloading them an expensive proposition.

Instead we do this (and please don't tell any operators of photo sharing services lest they revoke our API key): When we first add a tag to the Photo Commons tracker, the software uses the Flickr API to retrieve the first page of 100 photos belonging to the tag. If all of these 100 photos are unknown, we retrieve the next 100, and the next, and so on. This ensures that on the first run we retrieve all the tagged photos.

On subsequent runs of the Flickr API client there is likely to be less than 100 new photos so on average we only end up retrieving one page for each run. For those 100 photos we check if the title, description and size has changed (for example if the photo has been rotated) and update our database if that's the case.

There are a couple of problems with this. Retrieving information about 100 photos still costs something in bandwidth. We could reduce this number, but then we'd be more likely to need additional pages for each client run. Also, we don't catch changes in photo information for photos after the newest 100; we could solve this by retrieving all photos for each run (or every day), but this would be really expensive.

SSE is a relatively simple solution to some of these solutions. With SSE we have a way of asking for a single RSS feed of all photos with a given tag. On subsequent runs we could simply ask for new photos and any previously fetched photos that have changed. We could notify the service of photos from other services that are new or have changed. And we could tell our clients exactly where photos come from and what revision number we have.

Post in comments if I've got this wrong and if you have other use cases.

tags: sse rss

Update: Here's a nice diagram.