Flow *

So only 2 days after my presentation at Xtech 2007 about user generated pipelines and how Microsoft have got something in store in this area. Microsoft release details of Popfly,

Popfly is the fun, easy way to build and share mashups, gadgets, Web pages, and applications.

There is a screencast which shows pretty much everything you can do at a basic level with Popfly. There's also some more focused videos here.

The service is split into two, one a application the other a service.

1. Popfly Creator is a set of online visual tools for building Web pages and mashups.
2. Popfly Space is an online community of creators where you can host, share, rate, comment and even remix creations from other Popfly users.

It looks good and works well. Almost anyone power user will get the hang of it within minutes but there is almost enough to keep more advanced users going for a while. However it falls down in the same places as Yahoo Pipes. No access to the local file system again. Theres even bigger problems when you compare it to my core principles of user generated pipelines.

• Definable
• Graphical
• Standard
• Shareable
• Open
• Non-proprietary

Popfly only manages to get Graphical and Sharable right. This is worrying but its still in Alpha, so who knows what might happen in the next version. Till then, there is a blog for the team and a few screenshots even.

I have uploaded my presentation, pipelines: plumbing for the next web fresh from the first day of Xtech 2007 today to Slideshare.

Here's the abstract... and you can download the paper here.

APIs are a great way of developers and hackers being able to access data and content from one provider. But with the trend of the “mash-up” has come the ability to join two or more providers together to the benefit of the user rather than the providers. This level of interoperability means people can start offering automation and new business opportunities by “chaining” services together. As many of us look towards the social benefits of a somewhat provider centralised Web 2.0, its easy to see how our single provider habits can be broken by user generated pipelines.

Like Unix Pipelines, a user generated pipeline defines how content and data goes through a series of pipes. Unlike UNIX pipelines these pipes can be a series of remote or local web-services, applications, transformers. A simple example could be, uploading a photo from your mobile phone to Flickr.com, then that same photo magically appears on your friends doorstep processed, nicely cropped with a related personal message with no more time or effort required from yourself. That sounds like magic to a user but is perfectly workable to a developer with enough time.

The Flickr example I gave works on an application being authorised to access a certain picture on Flickr. Flickr already has this feature in its API and many other services use this to provide services to there users. So in this example Preloadr.com are instructed to receive the picture and do the default image enhancement which there famous for. After the preloadr is finished the picture is passed on to delivr.net which can create postcards and send them to a person on request.

The developer would normally use multiple available APIs to create this task. Those services and applications which are not available will struggle to survive in the new ecosystem. Open services would be happy with the traffic and the business. The user would be happy with the time saving and convince. So ideally everyone will be happy.

This is not a new concept but how we manage this has existed in the domain of Apple-scripter’s, Perl and Python hackers. Automator by Apple is an example of this, but fails to gain momentum due to its proprietary nature to the OSX operating system.

Touchstone (currently in private alpha) is a next generation desktop RSS aggregator which has the concept of filters which can trigger things to happen like a email to a friend or an automatic post to del.icio.us. It has many plugin’s and the ability to add more, but its uncertain if the filters can be shared and are non proprietary.

I’m proposing that a series of pipelines will be ultimately definable, non-proprietary and shareable by anyone who can install and run a browser. If correctly executed a whole eco-system could grow out of this decentralised user driven behavior. I believe XML Pipeline Definition Language is the key to making this happen, but currently lacks vision.

The general view is that the presentation went down well and made sense. However I think people really wanted to see something which worked instead of slideware.

Thanks to Tom Morris, I'm left wondering what on earth Microsoft are working on. Hopefully it will be desktop pipes.

Microsoft is developing a tool that will allow non-programmers to customize and mash-up various Web 2.0 applications and services, say sources close to the company.

That tool — now code-named "Springfield," according to one source — is similar in concept to the recently introduced Yahoo Pipes composite-mashup tool introduced by Yahoo in February. Pipes provides a graphical-user-interace-based interface for building applications that aggregate Web feeds and other Web services.

While I can't confirm this for a fact, I have strong suspicions that "Springfield" is the new codename for the technology formerly known as Microsoft "Tuscany."

The Microsoft Tuscany codename first surfaced over a year ago, just after Microsoft Chief Software Architec Ray Ozzie proclaimed that all Microsoft products, going forward, will have some kind of services and/or Web 2.0-centric component. Tuscany was known to somehow be connected with Microsoft's push to enlist more nonprofessional programmers and hobbyists in its developer ranks. Microsoft subsequently released a number of "Express" versions of its developer and database products, targeted specifically at non-professional programmers. But to date, company officials have declined to discuss Tuscany details.

When I tested a whole load of desktop search engines I settled on Google desktop search because you could either scrape the well formed HTML or run a bat script which produced XML. However there were a few semantic engines which I couldn't get working. One I missed was Beagle ++ which thanks to Tom I'm looking at now. I've always liked Beagle since seeing it running somewhat live at Xtech 05 but never went with the Windows Port. Now I can go install it and the ++ version on my desktop.

Something which I never wrote about in my xtech paper is the semantic desktop.

I was searching for the new Ajaxy Xbox media centre web interface, but came across documentation for the Xbox media centre's HTTPAPI. Which means I can completely control my xbox via a pipeline interface. However there are issues.

1. Its all HTML
2. Its not valid HT ML
3. It seems a little temperamental on Action commands

For example here's how to get what the Xbox is playing right now.

http://xbox/xbmcCmds/xbmcHttp?command=getcurrentlyplaying

But it comes back like this.

<html>
<li>Filename:smb://stratrix/downloads/podcasts/The 1UP Show/041307.m4v
<li>SongNo:0
<li>Type:Video
<li>Title:041307.m4v
<li>Thumb:defaultVideoCover.png
<li>PlayStatuslaying
<li>Time:00:02:40
<li>Duration:00:43:56
<li>Percentage:6
<li>File size:475954023
</html>

Although this is nasty, its still useful. How many media devices under your TV have some kind of API? How many devices around our house support some addressable API?

I meet the Yahoo pipes team recently at Etech 07 and gave them a heads up about using there already established Yahoo Widgets to create the perfect flow setup. There widget engine already has access to pretty much everything on your desktop and it has an authenticated path back to the main server via the Y! Login system. It also has many APIs hooks into different desktop applications such as Outlook, iTunes, Winamp, along with your standard File system, System, etc. Another benefit of Yahoo Widgets is that its also running all the time so you can schedule pipes to start when ever you like. Now hows this for an example Yahoo!

A Pipeline which looks in my outlook calendar for birthdays. When there is one within 3 days of the date, it will look through Flickr for that person and create a custom e-card ready for me to review and hit send. Even if you left out the search through Flickr, it would still be a nice pipe to have.

I didn't even mention what crazy things could be done with Yahoo Desktop search or the Attention type things which could be done with the Yahoo Toolbar. Guys got to have his secrets.

I did also raise the question of Yahoo Pipes using XPROC or some other standard to export pipelines from Yahoo pipes to something else. It was obvious this was not ever really thought about, but they seemed keen to make the underlining xml for the pipes available soon.

Its honestly amazing that Yahoo have not taken the lead on this already. They have all the pieces in the right place but are stalling for some reason. Maybe they need more people working on this?

Bradley Horowitz did a good job of reminding the Pipes team that pipes and pipelines are exciting and maybe some of the most interesting stuff Yahoo!'s done in years. I have

Unfortunately if Yahoo! Were to roll out a flow system. It would be mainly proprietary and non-standard, which goes against the flow idea really.

I put it to you all like this, Amazon.com are well ahead of the pipeline game. Microsoft, Google and even Yahoo have got a lot of ground to cover. So why am I saying this?

I was sitting in the crowd typing up notes and a blog post (video) for the future of webapps. And hit me, wait a second. Amazon have built a bunch of services which are totally useable remotely and could work very well in a pipeline. On top of that, they have a perfect revenue model (charging per use not a subscription cost). So you can decide to use them within a few minutes and be setup and ready to go as quickly. All you need is a Amazon account.

• Amazon Simple Storage Service - S3
• Amazon Elastic Compute Cloud - EC2
• Amazon Mechanical Turk (Beta) - Turk
• Amazon Simple Queue Service - SQS

In the above diagram (yep I love Inkscapes new features), you have media going into an encoder. The encoder could be on your desktop or some external service. On the left you got free services like hey!watch and on the right Amazon EC2 which can be setup for encoding but at a bigger cost. Amazon EC2 would also be much quicker and more reliable that hey!watch. Depending on your needs you can choose either one. The Pay pipe would be faster and more reliable, while the Free pipe not so reliable and much slower. Once you've encoded, its a matter of where do you store the footage for viewing. Like before you can use the left hand side for free solutions or the right for costly but that reliability (good quality of service).

I had a chat with Werber Vogels after his presentation and he made it clear that the services such as S3 and EC2 could be used in a pipeline but Amazon are not interested in that end of a potential system. There expecting someone else to build that end. There only interested in providing the servers and infrustrtuction to make allow others to build systems. I did ask him if Amazon were planning to provide services on top of EC2 (for example encoding services) and S3 (like Webdav, CalDav) but he made it clear that would be left up to 3rd parties. So in my example, Hey!Watch could offer a pro version which is actually using EC2 instead of their own machines. Last of all he mentioned Amazon SQS could be potentionally very interesting for pipeline applications.

Bill Thompson has been thinking about Pipelines too. His post is sparked by Yahoos Pipes but he also thinks about the bigger context of pipelines. Here's some key quotes which I found very interesting.

We have had mashups for a while now, like the projects coming out of the BBC’s own Backstage project, but they generally require some programming ability and usually combine two sources at a time, like the BBC News and Google Maps mashup in Ben O’Neill’s excellent news map. Pipes take things much further. This isn’t user-generated content, it’s user-controlled content. And unlike personalised pages or simple feed subscriptions it really does put control into the hands of the user. Pipes mark the point at which remixing online content and creating mashups becomes something that anyone can do. If you can describe what you want then you can build it. It is also, of course, a collaborative environment, at least for now. You can take someone else’s pipe and ‘clone’ it to make your own, with no hint that there could be copyright or intellectual property issues here.

The idea of sharing pipelines is essential, without this your back in the world of programming. Because your just chaining things together, realisticlly there shouldn't be any copyright issues ever. This is another reason why I really think this should be a XML format aka not some propitery binary format. Allow anyone to just write it if they like but also allow others to build fluid interfaces for user-controlled or user-generated pipelines.

What was waiting for me in my inbox today...

To: Ian Forrester

We are pleased to accept the following proposal for XTech 2007.

• Pipelines: Plumbing for the next web

It has been scheduled for 16:45 on 16 May 2007.

Please confirm that you have received this acceptance and can deliver the presentation.

Thank you,
Edd Dumbill

So my presentation at BarCampLondon2 will be a very early draft for whats to come in May.