Personal Question

Question : Hey Phil, do you actually do any programming these days?

Answer : Yes. Quite a lot at the moment. Though it's a bit all over the shop.

I'm dipping a toe into Android programming. (And, hmmm ... Java .... I thought I'd got over my Java hangups by doing a lot of Processing, but it turns out that Processing just hides the crap and Android doesn't. Why hasn't Google picked up on Processing to turn it into a first-class Android art / game app. development environment?)

I'm mainly writing CoffeeScript. Some stuff related to my ongoing 3D modelling / desktop manufacturing projects. (Did I forget to mention those? I'm sure there's a half-written blogpost somewhere.) Some work towards an SdiDesk-derived network diagramming plugin for Smallest Federated Wiki (held up by silly problems). Some other bits and pieces. I've recently been playing with Jison, which rocks. And I'm about to investigate angular.js which looks pretty good.

There's a project for small stand-alone web-servers that I'll talk about more if / when it takes off.

I've been trying to compile example VST instruments  (C++) for some of my work with the Brasilia Laptop Orchestra, but it's driving me crazy. (I may go back to Pure Data which can be embedded in a VST.)

A bit of PHP, just simple small web-services.

I'm going to be teaching an Arduino course soon. So I'll be writing a bit of C and I want to try Occam-Ⲡ.

I'm still writing Python too. Mainly for short file transformation scripts or to prototype algorithms that later get translated into CoffeeScript.

Some of this stuff is headed for GitHub soon.

Quick SFW Generator

Smart Disorganized Individual philosophy is about doing things piecemeal; engaging in small opportunistic actions as and when the inspiration strikes. Today was just such a quick burst with the Smallest Federated Wiki.

I love SFW a lot, but it's a bit of a pain to bring a page or large chunk of writing from elsewhere into it. Adding one paragraph at a time is a nuisance.

So here's a quick tool to create an SFW page from an arbitrary large, multi-paragraph chunk of text. Just paste your text into the box, add a title and choose whether you want ordinary paragraphs (of the kind that most SFW installations use) or whether you want "wikish" (the UseMod derived format which is somewhat compatible with UseMod / SdiDesk markup). Hit the submit button, and it will deliver a json file suitable for dropping into the pages directory of your SFW installation.

Update : the source-code for this is part of Project ThoughtStorms on GitHub.  (Note: I may have broken other import scripts in the refactoring to make this import script work. Please log an issue if you find this.)

Project ThoughtStorms Goes Live

ThoughtStorms has been ported across to the Smallest Federated Wiki.

Of course, there are plenty of failures, missing pages, bits of markup that aren't handled correctly etc. But this is wiki, right? It's never perfect. It's always living and dying and composting. The problems will get flushed out, or worked-around, or become charming ruins over time.

That doesn't mean Project ThoughtStorms is over. It's barely even got started. There are more wikis to port. There's an SdiDesk converter to write. There's plenty of gardening that I am committing to doing.

And then there's the really exciting stuff ... ;-)

Project ThoughtStorms

As mentioned previously, I've been looking into Ward Cunningham's "Smallest Federated Wiki" concept. And I'm increasingly impressed.

So much so, that I've re-oriented a lot of my projects around it.

What do I mean? A decent follow-up to SdiDesk has been promised for an embarrassingly long time. Over the years I've struggled with exactly what it should be and how it should be implemented. Largely whether it should be a desktop application or something you access via the browser. The browser has always been the logical answer but, until recently, the network diagramming aspect of SdiDesk was not really an option in mainstream browsers. OTOH, desktop GUIs open a can of worms. Which OS? Which GUI framework? How do I write installers and distribute? (And, frankly, what is my, as a non-Mac owner / developer, attitude towards the iPad?)

In 2012 though, HTML5 and CoffeeScript have become extremely plausible options for the client. And the server can become a simple wrapper around a basic PageStore. That's an architecture I've been meaning to get down to write. But it's the architecture that already exists for the SFW.

So, great! By hooking onto that project, I get my basic server / PageStore / client architecture free.

Furthermore it's extensible via plugins. So I can embed special types of paragraph data and special renderers. That's exactly what I wanted to do with the new SdiDesk - instead of having *pages* that were network diagrams or grids, have these as individual components of pages. This is perfect. I can concentrate on what interests me - the special plugin types - and Ward's team can do the infrastructure. :-)

Not to mention, Ward and co. are doing amazing plugin wizardry already : hooking data-feeds from Arduinos, graphing it, bytebeats, calculators. It already has a lot of what looked nice about QEDWiki.

The multi-panel view surprised me initially, but it's really useful for refactoring. And that's going to help me considerably with wiki-composting.

Finally, the "federated" part of the Simplest Federated Wiki is the answer to a bunch of problems I didn't even know I had. Or, at least, didn't conceptualise well. How do I have a private wiki (like a local SdiDesk, where I like to draft things before they go public) AND a public wiki (like ThoughtStorms) and make it easy to move newly public stuff from one to the other? How do I balance the desire to have special project focused wikis (like the OPTIMAES one) with wanting to refer to that stuff from the main wiki? How do I balance contributing to my own wiki and contributing to other communities' wikis?

So, I'm sold. As Dave Winer likes to say, it's the second mover who makes the standard. And that's what I want to help with. There's enough overlap between the SFW and the things I've been wanting to do over the last few years that it makes sense for me to implement my ideas as plugins for the SFW, to port my wikis over to to it and to go around shouting about how wonderful it is. Because, actually, it is pretty damned wonderful.

So, Project ThoughtStorms is where I'm putting the code: so far, converters from the ThoughtStorms UseMod and the SdiDesk formatted pages, and plugins to render the markup. I'll be porting ThoughtStorms over to a SFW server soon. Then I'll be doing some serious refactoring and cleaning up the actual writing. Trashing a lot of the ephemeral junk and dead-links. TS has become a bit of a museum, which it shouldn't be. It should be a living, learning, and forgetting thing.

After that, I'll be sitting down to do some of the other things I've wanted to do in a wiki context but not had the platform to do justice to.  Now I think I have one.

Import question. Is anyone currently using SdiDesk? Write in the comments if you are.

Not making as much progress on SdiDesk.NET as I thought I would. How can Visual Studio 2008 be soooo S-L-O-W? I swear it takes between 30 seconds and minute both to start and stop(!!!) my program executing.

At least with punched cards you knew where you were.

It's definitely SdiDesk week this week. Watch the repository.

Quick answer to the guy who used yesterday's Form Experiment to ask what's happening with SdiDesk ...

Here's the status report :

1) In the last couple of weeks I quit my job and moved back to the UK ... which has been taking up quite a lot of my time and energy.

2) Once we've settled in, I hope I can get some time to focus on my projects ... including SdiDesk (and GeekWeaver, MTC etc.) The good thing, no more distractions from my previous demanding day-job. The bad thing, after a brief holiday I need to find work in the UK. (Offers, tips and suggestions are, of course, welcome)

3) SdiDesk was converted to VB.NET this year. And the source-code in progress is available on Google Code.

4) I am NOT a VB.NET programmer, and frankly, from the little I've played with it so far, I'm not very excited about getting more involved. I admired VB exactly because the combined language + IDE made Windows programming mindlessly easy. Throw away that virtue (as VB.NET seems to have done, and I'm blaming the new, incredibly sluggish and cluttered VS2008 as much as changes to the language) and it has little to recommend it against other languages.

5) On the hand, I'm a pragmatist and often able to find something interesting pretty much anywhere. I also know that in a recession I may not be able to be too fussy when it comes to getting a job. So I will be spending a bit more time over the next month or so tidying up the VB.NET codebase, fixing some egregious issues, and making an installer. It's going to be useful to me to be able to say that I can operate in the VB.NET world, and produce working products.

6) Longer term, my preference and commitment is still to a Python SdiDesk-like thing. And most-likely a Python server with UI in the browser (javascript etc.) I've cooled towards the idea of Flex (mainly because my trial copy expired and I remembered all the problems of depending on proprietory software)

The attraction of anything other than the browser has always been the vector drawing in the network diagramming part of the software. But I'm sure that if I just wait a little bit longer, the browser will eventually be able to handle this too.

7) As always, I'm not unaffected by user feedback and other things going on in my life :-) ... if there's suddenly a surge in interest or demand for a VB.NET SdiDesk then I may reconsider.

8) Joe Question asks : "how risky is it to commit myself to SdiDesk.NET then? What about all my pages?" Answer : SdiDesk.NET reads your existing PageStore files. There may be some issues with the size of the diagrams, but everything works. If it doesn't, tell me.

However, you can't even try SdiDesk.NET currently unless you're a VB programmer because it's only available in source form. There will certainly be a build for end-users this year and it will read your existing PageStore. The main reason you want this is if you're an existing SdiDesk user who has moved (or will move soon) to Vista where the old VB6 version won't run.

I'm always committed to upgrade compatibility. You'll be able to move your existing SdiDesk pages to SdiDesk.NET, and you'll be able to import them into a future Python version.

Surprising SdiDesk news!!!!!

Like Cthulhu, SdiDesk, is still, currently dead, but occasionally stirs in its sleep and sends out weird dreams to the minds of men. Here's one-such ...

Around this time last year I got a new laptop with Vista and discovered that the old SdiDesk (in VB6) didn't work. Not having VB6 on the machine (or any installers for it) I didn't have a way to fix the problem. And anyway, I was (am) emotionally committed to getting off the Microsoft / VB treadmill and moving to a Python SdiDesk. (Really, I am!)

Six months later, though, heavily involved with GeekWeaver etc., I realized that I'd done nothing towards it. And SdiDesk was still broken.

I came across the new Visual Studio 2008 Express (free-as-in-beer) edition and decided to take a quick look.

It was s-l-o-w even on this newish laptop. And heavy, and I couldn't make much of it. But I realized that a) while I still wanted nothing to do with VB.NET and Windows-only programming b) SdiDesk was going to go extinct pretty quickly if it couldn't run at all. There seemed to be a closing window of opportunity to keep the original code-base and program alive. And I couldn't quite bring myself to let it die entirely.

So why not find someone else, who already knows about his kind of thing (VB6 to VB.NET conversions)? Via Rentacoder I contracted Zebo in Faisalabad, and thanks to some dedicated work by him, the SdiDesk source is now converted to VB.NET.

I'm not entirely sure where I'm going with this. The job Zebo did was a straight upgrade of the original VB6 code, (using the mechanical conversion and manually fixing the things the update couldn't handle). It's taken a while to sort out some weird security issues (eg. why the hell wouldn't Vista let it see Today's date?)

What there is now is new source-code which I've put on Google. If you're a VB.NET programmer you can check it out of the SVN repository and it should run.

I want to make a couple of minor tweaks before I make an installer for end users.

After that I'm keeping an open mind. It will depend a lot on whether existing SdiDesk users upgrade and new users appear. In general, the aim is still to move off VB, but exactly how and when, is open-ended.

Remember that this is the official blog for SdiDesk news and discussion so you can always get the latest news here.

hat-tip Bill Seitz : Dr. Drang builds a python-based desktop-like notebook.

Dan Bricklin :

Socialtext is announcing today that they are adding integrated spreadsheet capability to their enterprise-level wiki, making use of the new SocialCalc code I've been developing with them. This isn't just a repository of separate spreadsheets, nor a separate standalone system like wikiCalc, but rather a full wiki where a page can be either the traditional paragraphs of text or a spreadsheet grid.

Cool ... now, if they could just *also* add network diagramming as another page-type that would really be getting somewhere. (Not, of course, that the grid pages in SdiDesk really achieved "spreadsheet" status ... but that was always a long term hope.)

Ah ... well ...

Actually, there may be some SdiDesk news soon ... you never know.

Update : Bricklin has longer background piece.

Smart Disorganized Philosophy #1

I've been on a Smart Disorganized Individuals tip for several years, now. Always writing software compatible with that philosophy.

But what is the philosophy? What does this software mean?

In this series I'll start to make some specific notes towards that. Here's the first.

SdiDesk is wiki : a network of texts.

GeekWeaver is a programming language written in an outliner : a hierarchy.

Mind Traffic Control is a multi-user, dynamic queue, a "flow" of tasks.

Each is SDI. Each is completely different. Each is for specific purpose.

SdiDesk excels at capturing ideas and the relationships between them that are static.

Outlines excel at authoring or creating structure which is ultimately intended for a reader.

Flows excel at capturing change and movement.

OTOH, each is bad for something. These are true, even if you might imagine them not to be.

Wiki is surprisingly bad for authoring. Outliners are surprisingly bad for managing todo-lists. Flows are a surprisingly bad place to put ideas that you want to keep forever.

Wiki is great for writing, but awkward for the kind of reworking and structuring needed to polish a document for an external audience.

Outliners fail to match the dynamism of shifting tasks and priorities in the real world. They focus on making a structure of something which needs little structure.

MTC will lose your ideas when they are no longer in the future.

I'm way too busy now ... but I'm having some kicking ideas about my own grandiose bid to improve programming ...

... think GeekWeaver in an SdiDesk-alike editor (obviously).

Then imagine that *everything* is a template.

That's the way that GeekWeaver is already going. Everything is a template (ie. has named slots that can be filled). Function calls are just the injection of a data-block into that template. Another way of putting it, all objects know how to handle the "call" message with a data-block as argument, even if they don't do anything very useful with it.

But now imagine that all the types of things you can get in SdiDesk ... text pages, grids, network diagrams, are also templates. And you can plug and pipe them together any way you like. One page can hold a table, another a network diagram-shaped template, and a third can be specified as the result of injecting the first into the second.

Hmmm ... this definitely looks like it's going in the right direction.

Update : for people wondering how the whole new-SdiDesk-in-Adobe-Flex? thing is going. I solved something I thought was a problem yesterday.

I now have a (very fragile) Flex front end which can pass plain-text GeekWeaver programs to a web-server with GeekWeaver embedded, and get a compiled chunk of GeekWeaver out.

That's very cool ... unfortunately it also revealed yet more problems with the GeekWeaver compiler which need fixing before I can release the next version.

Still, it looks promising. The signs are better and better that something interesting is coming out in the next 3 months. :-)

Just spent the last few hours downloading and playing with the beta of Flex 3, Adobe's IDE for Rich Internet Applications (ie. applications running on the Flash Virtual Machine) which is based on Eclipse and has an XML-based UI / form description language more or less like HTML.

I'm having two thoughts about it. One is a kind of sigh of relief. This is, after all, finally, The One. After years of fruitless searching I'm pretty sure here's a framework I can settle down with and commit to, and start making babies with. At least, it's more or less mature enough, handsome enough and well endowed enough to put these thoughts into a girl's head. A browser and a desktop? Holidays in Windows, Mac and Linux? Own grid and canvas. Tabbed notebook and some cute chunky buttons.

And it's all done in a way that's pretty self-evident when you look at a few examples. Forms and input widgets are described in XML. They layout nice; and you can start banging them in and prototyping the look of your interface in a couple of minutes. The round-trip from coding to running and testing is a bit slow on my poor 512MB Vista laptop, but it's going to be bearable. And the Eclipsyness of it all is comfortingly familiar if a trifle overblown.

As long as my next experiments turn out right (the one where I try to find a tutorial example of pulling data off a server over http, and the one where I try to compile with AIR into a stand-alone application) then I'm sold.

But there's another part of me going, "huh? Is this all there is? WTF?"

I mean, it's 2007 and I'm happy because I've finally found a way to make GUIs that's sufficiently lower than my pain threshold that I might actually get a piece of software released again. Zowie! But that's what I had with Visual Basic 6 - which came out in about 1997!

In fact, I was already a Pythonista before I started writing SdiDesk in VB. And I only pulled out VB (a language I thought I'd left behind for good) because I got impatient to see what the UI of an SdiDesk could look like and thought I'd prototype it. As often happens, the prototype spiraled out of control as I kept thinking, "maybe I can just also add ... " and within a month or so it had already started to grow into a real program. Another phase of development with some serious refactoring and cleaning up the internal architecture, and it was a quite respectable and powerful bit of software (If I say so myself; I'm talking about the time I made the tutorial screen-casts.)

(Then, of course, I hit the crisis of not wanting to be on the Microsoft treadmill and forced to upgrade to .NET; even though it was obvious that VB6 was as extinct as a very extinct thing from the Lower Devonian period. But also of not having any viable alternative. )

So there's a sense that Flex smells extraordinarily similar. I can see how you can knock out your prototype interface and start building backwards from it. That feels good. That's why it seems like this is plausible option to get development rolling again.

But, like I say, it *is* basically what I had 10 years ago. Except with javascript dressed up in Java's suit and tie to look more grown-up and respectable. And XML.

In fact, I was gchatting to Zbigniew earlier today, and realizd that all of this stuff is hardly a big advance on Hypercard back in 1985. Or perhaps Smalltalk 1972. Why the hell haven't we progressed further? Why am I struggling on each new platform to rediscover the level of comfort I had on the previous one? What's going wrong here?

I suppose it could just be that the idea of quick GUI builders is inherent in the idea of a GUI?

Or maybe we programmers of the noughties need to get our acts together and start coming up with serious new, cool shit. Stuff which couldn't have been thought up in the 60s and 70s. Stuff which is radically easier and more productive than something we had 10 years ago. Something like reinventing Lisp with a cleaner (non)syntax ... erm ... well anyway, I'm off to do more experiments with Flex and try to get it to talk to some kind of server.

If I succeed, then expect to see some interesting developments along the lines I mentioned earlier today ... steps towards a new SdiDesk, possibly a GeekWeaver development environment ... maybe even the long fabled, but never released SystemSketch. Or even the more outré things I've got buzzing around in my fevered imagination like "SexyCells" and "FlowerBrush".

Of course, it still sucks that Flex / Flash seems to have no musical ability whatsoever so Gbloink! doesn't look like an option. Which is a double pity because I think it would make a great Chumby widget and that would have justified me buying one.

:-(

Quick Note : I just had a revolution in my thinking, triggered by Enso but influenced by several other recent trends.

You write Enso "extensions" as XML-RPC servers sitting on your local machine, register them with Enso and it calls them using XML-RPC. I tried the example from the tutorial and it's very cute and simple. So I've decided this may be a way forward for the dilemma which has kept the "new SdiDesk" on hold for several years(!!!)

If you remember, the issue has been whether the new SdiDesk should be a "web"-application (accessed through the browser) or a desktop application? And how to implement the UI.

The problem with the "browser" answer has always been : "but how to do the network diagramming bit?" - which requires interactive vector graphics. SVG doesn't seem quite stable or cross-browser enough. Canvas isn't cross-browser. Flash is proprietory.

The problem with the "desktop application" is, well, the many rival Python GUI libs with different degrees of maturity. And the question of getting them to work cross-platform.

A meta-problem : I just don't have time and energy to go through learning lots of different GUI frameworks to decide which I want to use. And the ideal answer to the browser / desktop question is probably "both" which doubles the amount of work.

In fact, one of the motivations for GeekWeaver is to see if I can use it define a higher-level UI description that can be compiled down into both XHTML and a GUI widget-set.

Of course, with XUL, Open Laszlo, Adobe Flex, XUML etc. lots of people have been looking at something like HTML for defining desktop apps. too, but the free Python frameworks don't seem to have caught up with that yet. XUL sounds most promising especially with the open sourcing of Active State's Komodo ... but that's also a big complicated thing to even start looking into.

Anyway, inspired by the Enso extensions and Bruce Eckel's interesting example of hooking up a Python XML-RPC server behind an Adobe Flex application I've been wondering about blowing the project apart entirely into a number of services connectd only XML-RPC or something more ReSTful.)

So there'll be a WADS (SdiDesk PageStore) service, a GeekWeaver interpreter service, a "user navigation history service" etc. Even the glue which ties all this together will be another service that's neutral about the user-interface. Effectively the model, view and controller will be completely separate programs. (I know, I know ;-) )

Then there'll be a variety of different types of access with different UIs.

Now that Adobe's AIR has put Flash on the desktop and made it a serious rival to the Java virtual machine - I *am* tempted to put some time into learning it. It would give me a reasonable GUI widget set (including TreeGrid, yay!) and the vector graphics needed for networks. It will run both on the desktop and in the browser and on Linux, PC and Mac. The tools are free-as-in-beer (Flex beta) or free-as-in-speech (Open Laszlo). It's also possible to imagine generating the Flex XML or the Open Laszlo format directly from GeekWeaver. (Aside : I wish ActionScript hadn't borrowed quite so much from Java, but still, I'd only be using it for the front-of-the-front-end UI stuff.)

Then I want to experiment with Enso access - for example, Enso commands like "page HelloWorld" or "pagehistory ChocolateCake"

And there'll be web-browser access probably through a "gateway" that accepts http requests and spits out HTML for a browser. The only thing I don't know is whether I *really* should be using WSGI somewhere here.

Anyway ... that's a quick update of my thinking this week ... tune in soon for the next GeekWeaver release (really, it's coming together, and gonna be very powerful). Then it's back to work on this larger project.