Flow *

I meet the Yahoo pipes team recently at Etech 07 and gave them a heads up about using there already established Yahoo Widgets to create the perfect flow setup. There widget engine already has access to pretty much everything on your desktop and it has an authenticated path back to the main server via the Y! Login system. It also has many APIs hooks into different desktop applications such as Outlook, iTunes, Winamp, along with your standard File system, System, etc. Another benefit of Yahoo Widgets is that its also running all the time so you can schedule pipes to start when ever you like. Now hows this for an example Yahoo!

A Pipeline which looks in my outlook calendar for birthdays. When there is one within 3 days of the date, it will look through Flickr for that person and create a custom e-card ready for me to review and hit send. Even if you left out the search through Flickr, it would still be a nice pipe to have.

I didn't even mention what crazy things could be done with Yahoo Desktop search or the Attention type things which could be done with the Yahoo Toolbar. Guys got to have his secrets.

I did also raise the question of Yahoo Pipes using XPROC or some other standard to export pipelines from Yahoo pipes to something else. It was obvious this was not ever really thought about, but they seemed keen to make the underlining xml for the pipes available soon.

Its honestly amazing that Yahoo have not taken the lead on this already. They have all the pieces in the right place but are stalling for some reason. Maybe they need more people working on this?

Bradley Horowitz did a good job of reminding the Pipes team that pipes and pipelines are exciting and maybe some of the most interesting stuff Yahoo!'s done in years. I have

Unfortunately if Yahoo! Were to roll out a flow system. It would be mainly proprietary and non-standard, which goes against the flow idea really.

Bill Thompson has been thinking about Pipelines too. His post is sparked by Yahoos Pipes but he also thinks about the bigger context of pipelines. Here's some key quotes which I found very interesting.

We have had mashups for a while now, like the projects coming out of the BBC’s own Backstage project, but they generally require some programming ability and usually combine two sources at a time, like the BBC News and Google Maps mashup in Ben O’Neill’s excellent news map. Pipes take things much further. This isn’t user-generated content, it’s user-controlled content. And unlike personalised pages or simple feed subscriptions it really does put control into the hands of the user. Pipes mark the point at which remixing online content and creating mashups becomes something that anyone can do. If you can describe what you want then you can build it. It is also, of course, a collaborative environment, at least for now. You can take someone else’s pipe and ‘clone’ it to make your own, with no hint that there could be copyright or intellectual property issues here.

The idea of sharing pipelines is essential, without this your back in the world of programming. Because your just chaining things together, realisticlly there shouldn't be any copyright issues ever. This is another reason why I really think this should be a XML format aka not some propitery binary format. Allow anyone to just write it if they like but also allow others to build fluid interfaces for user-controlled or user-generated pipelines.

What was waiting for me in my inbox today...

To: Ian Forrester

We are pleased to accept the following proposal for XTech 2007.

• Pipelines: Plumbing for the next web

It has been scheduled for 16:45 on 16 May 2007.

Please confirm that you have received this acceptance and can deliver the presentation.

Thank you,
Edd Dumbill

So my presentation at BarCampLondon2 will be a very early draft for whats to come in May.

By accident or carefully leaked?

Chris Saad posted a picture of his desktop which also shows off the new Touchstone beta interface.

When you see it like this it makes a lot of sense. But I've had to set 3 messages almost exactly the same 3 times over. I should be able to set it once and that be the end of it - surely?

This is a perfect example of what a user generated pipeline should be allowed to do. Either one of these are possible but only currently using lots of Applescripting or some clever serverside scripting

Input text in Twitter > Use Twitter API to pull current status into a temporary area (buffer/variable) > Use Skype API to update your status message. Or the reverse of this from Skype to Twitter using there API's.

Input text into 3rd party box > Apple script updates Twitter, Skype and any IM program you have running with the same message if there set to Away already

I wouldn't be surprised if Twitter added the ability to do this type of thing, but it would require you to input your username and password for skype or instant messenger user. This is currently how Blip.TV and Flickr.com can provide pipelines out to other providers. But this is not scalable, shareable and certainly not advisable.

Technorati Tags: api, im, interop, jabber, message, notification, presence, skype, twitter

I've listened to this podcast quite a few times now and theres a bit I've cut out and posted up on Blip because IT Conversations doesn't have the clip feature any more.

So generally there talking about the fact that Open source is full of developers who can actually build things. This is frustrating for others if you buy into the concept of OSS2 (open source society), where open source software is just one aspect of a much larger landscape. Human ordinated services like Ning.com, Flickr.com and Ebay.com, bring complex programming tasks to a much wider audience. Ning.com is a great example because it allows you to simply look at what someone else has done, copy it and modify it. But still Ning is still too programmer centric for there liking. Then someone talks about Excel Macros and how you didn't need to be a programmer or programmer minded to create Excel Macros. They could have been the first step over the programming interface problem for a lot of people.

Bingo! Excel macros are quick and easy to create and they were sharable with friends and workmates. You could learn more about the language behind the macros called VBA if you liked and a lot of people did. I know lots of programmers who hated VBA, some of them would go as far as to call it VB for girls even (which is actually a pretty bad and sad comment). But the fact is that it helped people get what they wanted to achive done, and you can't laugh at that.

I've mentioned Apple's Automator which is simply a macro for applescript. And to be fair they are kind of simular if you discount the fact Excel Macros only lived in the walled garden of Excel and the Office suite. Maybe that was the problem with them generally. But It certainly interesting and makes me think the answer to my question is yes, or at least user generated pipelines (Xproc or whatever) need some macro application and need to be at least as sharable as Macros in Excel. Otherwise we got no chance of non-programmers using them.

I have been reviewing some of the XProc information and draft specs, and came across Norman Walsh's overview of the Xproc 1st draft.

I first love the quote he uses.

Progress isn't made by early risers. It's made by lazy men trying to find easier ways to do something. - Robert Heinlein

Awesome that pretty much sums up everything. When I talk about pipelines, most people ask me why would a developer use a higher level pipeline language over simply writing the code. I always suggest the lazyness and time constraints put on people. Its going to be or should be simplier to write a pipeline that code the whole lot.

Norman uses this example to prove the point.

<p:pipeline xmlns:p="http://www.w3.org/2006/09/xproc"
            name="pipeline">

<p:declare-input port="document"/>
<p:declare-input port="schema"/>
<p:declare-input port="stylesheet"/>
<p:declare-output port="result" step="transform" source="result"/>

<p:step name="xinclude" type="xinclude">
  <p:input port="document" step="pipeline" source="document"/>
</p:step>

<p:step name="validate" type="validate">
  <p:input port="document" step="xinclude" source="result"/>
  <p:input port="schema" step="pipeline" source="schema"/>
</p:step>

<p:step name="transform" type="xslt">
  <p:input port="document" step="validate" source="result"/>
  <p:input port="stylesheet" step="pipeline" source="stylesheet"/>
</p:step>

</p:pipeline>

Can you tell me what that pipeline does? I bet you can, without even reading the specification: it performs XInclude processing on a document, validates it against a schema, transforms it with XSLT, and returns the result.

Norman Walsh also uses a few examples comparing Java vs Xproc. He concludes using this...

I think from an ease-of-use and programmer productivity point of view, pipelines are an obvious win: they're more declarative, they allow application behavior to be modified (within limits) without touching a line of code, and they potentially have much better performance. That last point is probably worth a little exploration. There are at least two ways in which using pipelines can lead to improved performance. One is a generality, if you're coding up your processing directly in Java (or C or Ruby or whatever), then optimization is your problem. If lots of folks are using pipelines then it makes sense to invest resources in improving the performance of the code that implements them. Making that code perform better automatically helps you (and everyone using pipelines). A less immediately obvious benefit arises from the fact that XProc has a fairly large vocabulary of built in steps. They aren't all spelled out in the current draft, but will eventually include steps to add, rename, and delete elements and attributes; process regions of a document based on XPath expressions, combine documents, split documents, extract content, inject content, etc.

A good arguement for the sharing non-proprietary notion which I'm calling for.

Of course the important word way back up there in the paragraph before the pipeline example was “standard”. None of this work is a real win for users unless they can reasonably expect interoperable implementations. I want (desperately sometimes) to be able to distribute pipelines with the same ease that I now distribute XSLT stylesheets.

I've been sitting on this blog post for bloody ages. But Tom's post has tipped me over.

Want to see something cool that's coming soon? Take a gander at XProc - the XML Pipeline Language. It's a way of defining a series of processes that operate on an XML file - for instance, running it through XInclude, schema validation, XSLT and making choices etc. It is great in as much as it's abstracting yet another layer out of the processing systems (SAX, DOM etc.) and their implementations (Java, PHP etc.) - obviously there are problems with that. Norman Walsh says that it's quite likely to be finished early next year. Kurt Cagle of XML.com thinks this is a good thing, and should fit in to the XML+REST ecosystem nicely.

So I've been thinking about some presentations and talks I'm planning on giving next year. I can't quite put my finger on the exact term but I know through blogging it and being very open about my thoughts I might reach a set of conclusions or at least points worthy of talking about with others..In my usual style a lot of the stuff is scattered around all over the place, so I'm going to try and use a wiki or something else to tie things togther.

My abstract for Etech 2007, which didn't get accepted.

API's are a great way of developers being able to access data and content from one provider. But with the trend of the mash-up has come the ability to join two or more providers together to the benefit of the user. This level of interoperability means people can start offering automation and new business opportunities by chaining services together. As many of us look towards the social benefits of a somewhat centralised Web 2.0, I can see how our single provider habits will be broken by the user generated pipelines. Like Unix Pipelines, a user generated pipeline can be used to send content through a series of pipes. But unlike UNIX pipelines these pipes can be a series of remote or local webservices, services, applications, transformers, etc. A simple example could be, uploading a photo from your mobile phone to Flickr, then that same photo magically appears on your friends doorstep processed, nicely cropped with a related personal message with no more time or effort required from yourself. Thats the magic of pipelines. This is not a new concept but how we manage this has existed in the domain of Apple-scripters, Perl and Python hackers. Automator by Apple is an example of this, but fails due to its proprietary nature. I'm proposing that a series of pipelines will be ultimately definable, non-proprietary and shareable by anyone who can install and run a browser. A whole eco-system will grow out of this decentralised user driven behaviour, which I call Web 2.5.

The Flickr example I gave works on an application being authorised to access a certain picture on Flickr. Flickr already has this feature in its API and many other services use this to provide services to there users. So in this example Preloadr.com are instructed to receive the picture and do the default image enhancement which there famous for. After the preloadr is finished the picture is passed on to delivr.net which can create postcards and send them to a person on request. This is all possible now with simple AppleScript or some other scripting language like Perl but requires a intimate knowledge of the scripting language. A user generated pipeline would be the higher level language to describe the Flickr example

Addy Santo of Santomania once wrote this quite fantastic application called Blogwave which he has not been updated for at least 3 years now. Its a multi purpose .net application which can consume RSS feeds (generator), transform them with some parameters like sort. It would then send them somewhere else, for example FTP, Email, SMB, etc in a RSS or Text form. What I found interesting about it was actually, it would create timed batch tasks in the standard Windows scheduler (something not many people use on there desktop). So in actual fact, it was a GUI for the command line in Windows. The application was a head of its time and unfortually not open source, so its kind of died but can still be used if you find the right link. But the concept is key, a GUI creates scripts or manages the complex pipeline process. The different pipes are already defined so you don't need some low level code to manage it. It seems Touchstone will take over from where Blogwave went, but I'm not on the alpha programme so I can't actually play with it.

I have tons of other examples but I'm now saving them up for the wiki and for my talk at Xtech 2007 which I'm currently rewriting my failed Etech proposal for right now.