Programming Language Features for Large Scale Software

My Quora Answer to the question : What characteristics of a programming language makes it capable of building very large-scale software?

The de facto thinking on this is that the language should make it easy to compartmentalize programming into well segregated components (modules / frameworks) and offers some kind of "contract" idea which can be checked at compile-time.

That's the thinking behind, not only Java, but Modula 2, Ada, Eiffel etc.

Personally, I suspect that, in the long run, we may move away from this thinking. The largest-scale software almost certainly runs on multiple computers. Won't be written in a single language, or written or compiled at one time. Won't even be owned or executed by a single organization.

Instead, the largest software will be like, say, Facebook. Written, deployed on clouds and clusters, upgraded while running, with supplementary services being continually added.

The web is the largest software environment of all. And at the heart of the web is HTML. HTML is a great language for large-scale computing. It scales to billions of pages running in hundreds of millions of browsers. Its secret is NOT rigour. Or contracts. It's fault-tolerance. You can write really bad HTML and browsers will still make a valiant effort to render it. Increasingly, web-pages collaborate (one page will embed services from multiple servers via AJAX etc.) And even these can fail without bringing down the page as a whole.

Much of the architecture of the modern web is built of queues and caches. Almost certainly we'll see very high-level cloud-automation / configuration / scripting / data-flow languages to orchestrate these queues and caches. And HADOOP-like map-reduce. I believe we'll see the same kind of fault-tolerance that we expect in HTML appearing in those languages.

Erlang is a language designed for orchestrating many independent processes in a critical environment. It has a standard pattern for handling many kinds of faults. The process that encounters a problem just kills itself. And sooner or later a supervisor process restarts it and it picks up from there. (Other processes start to pass messages to it.)

I'm pretty sure we'll see more of this pattern. Nodes or entire virtual machines that are quick to kill themselves at the first sign of trouble, and supervisors that bring them back. Or dynamically re-orchestrate the dataflow around trouble-spots.

Many languages are experimenting with Functional Reactive Programming : a higher-level abstraction that makes it easy to set up implicit data-flows and event-driven processing. We'll see more languages that approach complex processing by allowing the declaration of data-flow networks, and which simplify exception / error handling in those flows with things like Haskell's "Maybe Monad".

Update : Another thing I'm reminded of. Jaron Lanier used to have this idea of "Phenotropic Programming" (WHY GORDIAN SOFTWARE HAS CONVINCED ME TO BELIEVE IN THE REALITY OF CATS AND APPLES) Which is a bit far out, but I think it's plausible that fault-tolerant web APIs and the rest of the things I'm describing here, may move us closer.

Cthulhu

My software is more or less like Cthulhu. Normally dead and at the bottom of the sea, but occasionally stirring and throwing out a languid tentacle to drive men's minds insane. (Or at least perturb a couple of more recklessly adventurous users.)

However there's been a bit more bubbling agitation down in R'lyeh recently. The latest weird dream returning to trouble the world is GeekWeaver, the outline based templating language I wrote several years ago.

GeekWeaver was basically driven by two things : my interest in the OPML Editor outliner, and a need I had to create flat HTML file documentation. While the idea was strong, after the basic draft was released, it languished. 

Partly because I shifted from Windows to Linux where the OPML Editor just wasn't such a pleasurable experience. Partly because GW's strength is really in having a templating language when you don't have a web server; but I moved on to doing a lot of web-server based projects where that wasn't an issue. And partly, it got led astray - spiralling way out of control - by my desire to recreate the more sophisticated aspects of Lisp, with all kinds of closures, macros, recursion etc.

I ended up assuming that the whole enterprise had got horribly crufty and complicated and was an evolutionary dead end.

But suddenly it's 2013, I went to have quick look at GeekWeaver, and I really think it's worth taking seriously again.

Here are the three reasons why GeekWeaver is very much back in 2013 :

Fargo

Most obviously, Dave Winer has also been doing a refresh of his whole outlining vision with the excellent browser-based Fargo editor. Fargo is an up-to-date, no-comprise, easy to use online OPML Editor. But particularly important, it uses Dropbox to sync. outlines with your local file-system. That makes it practical to install GeekWeaver on your machine and compile outlines that you work on in Fargo.

I typically create a working directory on my machine with a symbolic link to the OPML file which is in the Fargo subdirectory in Dropbox and the fact that the editor is remote is hardly noticable (maybe a couple of seconds lag between finishing an edit and being able to compile it).

GitHub

What did we do before GitHub? Faffed, that's what. I tried to put GeekWeaver into a Python Egg or something, but it was complicated and full of confusing layers of directory.  And you need a certain understanding of Python arcana to handle it right. In contrast, everyone uses Git and GitHub these days. Installing and playing on your machine is easier. Updates are more visible.

GeekWeaver is now on GitHub. And as you can see from the quickstart guide on that page, you can be up and running by copying and pasting 4 instructions to your Linux terminal. (Should work on Mac too.) Getting into editing outlines with Fargo (or the OPML Editor still works fine) is a bit more complicated, but not that hard. (See above.)

Markdown

Originally GeekWeaver was conceived as using the same UseMod derived wiki-markup that I used in SdiDesk (and now Project ThoughtStorms for Smallest Federated Wiki). Then part of the Lisp purism got to me and I decided that such things should be implementable in the language, not hardwired, and so started removing them. 

The result was, while GeekWeaver was always better than hand-crafting HTML, it was still, basically hand-crafting HTML, and maybe a lot less convenient that using your favourite editor with built-in snippets or auto-complete.

In 2013 I accepted the inevitable. Markdown is one of the dominant wiki-like markup languages. There's a handy Python library for it which is a single, install away. And Winer's Fargo / Trex ecosystem already uses it. 

So in the last couple of days I managed to incorporate a &&markdown mode into GeekWeaver pretty easily. There are a couple of issues to resolve, mainly because of collisions between Markdown and other bits of GeekWeaver markup, but I'm now willing to change GeekWeaver to make Markdown work. It's obvious that even in its half-working state, Markdown is a big win that makes it a lot easier to write a bigger chunks of text in GeekWeaver. And, given that generating static documentation was GeekWeaver's original and most-common use-case, that's crucial.

Where Next?

Simplification. I'm cleaning out the cruft, throwing out the convoluted and buggy attempts to make higer-order blocks and lexical closures. (For the meantime.) 
  
Throwing out some of my own idiosyncratic markup to simplify HTML forms, PHP and javascript. Instead GW is going to refocus on being a great tool for adding user-defined re-usable abstractions to a) Markdown and b) any other text file.

In recent years I've done other libraries for code-generation. For example, Gates of Dawn is Python for generating synthesizers as PureData files. (BTW : I cleaned up that code-base a bit, recently, too.)

Could you generate synths from GeekWeaver? Sure you could. It doesn't really help though, but I've learned some interesting patterns from Gates of Dawn, that may find their way into GW.

Code Generation has an ambiguous reputation. It can be useful and can be more trouble than it's worth. But if you're inclined to think using outlining AND you believe in code-gen then GeekWeaver is aiming to become the perfect tool for you.

Java Hater

Someone on Quora asked me to answer a question on my personal history of using Java. It became a kind of autobiographical confession. 

I've never had a good relationship with Java.

My first OO experience was with Smalltalk. And that spoiled me for the whole C++ / Java family of strongly typed, compiled OO languages. 

Because I'd learned Smalltalk and this new fangled OO thing when it was still relatively new (in the sense of the late 80s!) I thought I had it sussed. But actually I had very little clue. I enthusiastically grabbed the first C++ compiler I could get my hands on and proceeded to spend 10 years writing dreadful programs in C++ and then Java. I had assumed that the OOness of both these languages made them as flexible as I remembered Smalltalk to be. I thought that OO was the reason for Smalltalk's elegance and that C++ and Java automatically had the same magic. 

Instead I created bloated frameworks of dozens of classes (down to ones handling tiny data fragments that would have been much better as structs or arrays). I wrote hugely brittle inheritance hierarchies. And then would spend 3 months having to rewrite half my classes, just to be able to pass another argument through a chain of proxies, or because somewhere in the depths of objects nested inside objects inside objects I found I needed a new argument to a constructor. The problem was, I was programming for scientific research and in industry but I hadn't really been taught how to do this stuff in C++ or Java. I had no knowledge of the emerging Pattern movement. Terms like "dependency injection" probably hadn't even been invented. 

I was very frustrated. And the funny thing I started to notice was that when I had to write in other languages : Perl, Javascript, Visual Basic (Classic), even C, I made progress much faster. Without trying to model everything in class hierarchies I found I just got on and got the job done. Everything flowed much faster and more smoothly.

Perl's objects looked like the ugliest kludge, and yet I used them happily on occasion. In small simulations C structs did most of what I wanted objects to do for me (and I did finally get my head around malloc, though I never really wrote big C programs). And I had no idea what the hell was going on with Javascript arrays, but I wrote some interesting, very dynamic, cross browser games in js (this is 1999) using a bunch of ideas I'd seen in Smalltalk years before (MVC, a scheduler, observer patterns etc.) and it just came out beautifully. 

It wasn't until the 2000s that I started to find and read a lot of discussions online about programming languages, their features, strength and weaknesses. And so I began my real education as a programmer. Before this, a lot of the concepts like static and dynamic typing were vague to me. I mean, I knew that some languages you had to declare variables with a type and in some you didn't. But it never really occurred to me that this actually made a big difference to what it was like to USE a language. I just thought that it was a quirk of dialect and that good programmers took these things in their stride. I assumed that OO was a kind of step-change up from mere procedural languages, but the important point was the ability to define classes and make multiple instances of them. Polymorphism was a very hazy term. I had no real intuitions about how it related to types or how to use it to keep a design flexible.

Then, in 2002 I had a play with Python. And that turned my world upside-down.
For the first time, I fell in love with a programming language. (Or maybe the first time since Smalltalk, which was more of a crush).

Python made everything explicit. Suddenly it was clear what things like static vs. dynamic typing meant. That they were deep, crucial differences. With consequences. That the paraphernalia of OO were less important than all the other stuff. That the fussy bureaucracy of Java, the one class per file, the qualified names, the boilerplate, was not an inevitable price you had to pay to write serious code, but a horribly unnecessary burden.
Most of all, Python revealed to me the contingency of Java. In the small startup where I'd been working, I had argued vehemently against rewriting our working TCL code-base in Java just because Java was OO and TCL wasn't. I thought this was a waste of our time and unnecessary extra work. I'd lost the argument, the rewrite had taken place, and I hated now having to do web-stuff with Java. Nevertheless, I still accepted the principle that Java was the official, "grown up" way to do this stuff. Of course you needed proper OO architecture to scale to larger services, to "the enterprise". Ultimately the flexibility and convenience of mere "scripting" languages would have to be sacrificed in favour of discipline. (I just didn't think we or our clients needed that kind of scaling yet.) 

What Python showed me was we weren't obliged to choose. That you could have "proper" OO, elegant, easy to read code, classes, namespaces, etc. which let you manage larger frameworks in a disciplined manner and yet have it in a language that was light-weight enough that you could write a three line program if that's what you needed. Where you didn't need an explicit compile phase. Or static typing. Or verbosity. Or qualified names. Or checked exceptions. What I realised was that Java was not the inevitable way to do things, but full of design decisions that were about disciplining rather than empowering the programmer. 

And I couldn't stomach it further. Within a few months of discovering Python I had quit my job. Every time I opened my machine and tried to look at a page of Java I felt literally nauseous. I couldn't stand the difference between the power and excitement I felt writing my personal Python projects, and the frustration and stupidity I felt trying to make progress in Java. My tolerance for all Java's irritations fell to zero. Failing to concentrate I would make hundreds of stupid errors : incompatible types, missing declarations or imports, forgetting the right arguments to send to library methods. Every time I had to recompile I would get bored and start surfing the web. My ability to move forward ground to a halt.
I was so fucking happy the day I finally stopped being a Java programmer.

Postscript : 

1) Something I realized a while after my bad experience was how important the tools are. My period in Java hell was trying to write with Emacs on a small-screen laptop without any special Java tools (except basic Java syntax colouring). I realize this is far from the ideal condition to write Java and that those who are used to Eclipse or IntelliJ have a totally different experience and understanding of the language. 

2) A few years later, I taught the OO course in the local university computer science department. All in Java. By that time, I'd read a couple of Pattern books. Read Kent Beck's eXtreme Programming. Picked up some UML. And I had a much better idea what Polymorphism really means, how to use Interfaces to keep designs flexible, and why composition is better than inheritance. I tried to get the students to do a fair amount of thinking about and practising refactoring code, doing test driven development etc. It all seemed quite civilized, but I'm still happy I'm not writing Java every day. 

3) A couple of years ago I did do quite a lot of Processing. I was very impressed how the people behind it managed to take a lot of the pain of Java away from novice programmers. I wonder how far their approach could be taken for other domains.

Why Pascal is Not My Favorite Programming Language

This is a great essay on what's wrong with Pascal. But really, it's a great essay on what are some of the nice touches of C that makes it such a good language.

Dog : A Social Language

Dog seems to be a little language for writing social software.

Initial thoughts :

Big question is what it compiles to. It's about time we had a programming language that compiles a single program down to parts that run on both server and clients, in a really easy and transparent way.

Building in knowledge of protocols like http and json and making services like twitter at least members of the standard library is a good idea.

Like most programmers, I'm sceptical of the  "easy English-like" nature of it. We've had plenty of time to learn that what's hard in programming is the logical thinking not the strange syntax. (Update : See my Quora answer)

But if Dog can become a little-language which makes it easy to configure and administrate social software back-ends then it will be very useful. Particularly if there are ways of compiling the same program down to multiple back-ends (Django, Rails, Chicago Boss etc.)

Nathan's University (Update)

Just to note, I think this course is pretty good : http://nathansuniversity.com/pl101.html

Make Your Own Programming Language

Nathan's University offers an online course.

I've signed up.

Bret Victor's Code / Drawing IDE

Excellent video

I'll have more to say and think about this.

RIP John McCarthy

John McCarthy has died.

Discussion on Lambda the Ultimate.

RIP Dennis Ritchie

Forget Steve Jobs, the guy who invented C has died!

Google's Dart

So Google's Javascript replacement language, Dart goes public.

Looks awfully like Java with a smattering of CoffeeScript. I like the empty compact constructor and the one-liner functions. But I'm not sure what those colon ones are doing.

Presumably some jQuery-like action with the document.query().

Looks a little bit messy, but then Javascript has got kind of messy. Shame they didn't try to go for the CoffeeScript cleanness.

In a sense, it may be rather similar to writing Processing if they produced a decent IDE.

On the whole, I think I can live with it.

Some interesting evaluation at Lambda the Ultimate.

Lunascript looks very interesting.

Why Programming Languages?

How could I have missed this guy commenting on Oli's Semantic Programming?

Dunno ... but I did.

Urbi is a great, parallel, event driven language for programming robots.

Watch the tutorial video.

Absolutely packed with interesting control structures to handle the implicit parallelism and event-driven nature of the language. Several important ideas : free subsumption architecture (you actually run many different programs in parallel, each dealing with certain motors and sensors, but interacting with each other only through the body of the robot; "blending" modes which let different programs send multiple instructions to the same motor at the same time; "tags" which let you interact with (start, pause, stop) running processes by name.

I like the look of the Clay programming language. C with variants.

How to write an interpreter in one day.

Now this guy is impressive.

Prediction I made :

I predict that in five years time, the hot language will either be a version of Python which has adopted ideas (Pattern matching arguments, Monads, immutability etc.) from Haskell; or a Haskell-like language which has borrowed some syntactic sugar from Python.

Don't know if I entirely believe that, but it's a good conversational gambit. And there's a certain logic to it if you look at the languages visually.

I'm personally, more into investigating Erlang at the moment, mainly because it seems more practical. The fast, parallel virtual machines are here and ready to go. But I could see myself looking back at Haskell too if it seemed to be gaining momentum.

I guess I also like the Erlang story on concurrency through message passing rather than transaction memories. And perhaps the whole typing thing is overdone in Haskell, although I can certainly see that it buys you more in that context than in, say, an OO language.

But I'd really love to see someone come up with mashup of the best bits of Python, Erlang and Haskell. It would *look* more or less like Python, although would find some-way to do multi-line lambdas etc. Like Haskell and Erlang it would have pattern matching arguments, functions defined over several definition statements, and immutable state. Like Erlang there'd be processes and messages as standard. Like Haskell, monads. Classes might be more like Haskell.

How might a language like this look?

 class Tree(val,left,right)
 def size(None) : 0
 def size(Tree(_,left,right)) :
    1 + size(left) + size(right)

Note that what's in brackets after the word Tree is neither a super-class (as in Python) nor a list of types as in Haskell nor a definitive list of field names. It's something more like information that a Tree is a tuple of three elements. And that by default they're called val, left and right.

These default names for the elements of the tuple can be over-ridden in functions which deconstruct a Tree using patterns. For example, the size function doesn't care about the initial val element and so leaves it blank.

I just thought of trying to make size a method of the class Tree. The syntax might then look something like this :

 class Tree(val,left,right) :
    def size(Tree(_,left,right)) : 
       1 + left.size() + right.size()

Where, obviously, the object itself is still (as in Python) explicitly passed as the first argument to the method, but instead of being called "self" we pick it up with the pattern of another Tree constructor and so it gets broken into _,left and right.

But where do we put the None / empty tree clause? My first thought was something like this :

 class Tree(val,left,right) :
   def size(None) : 0
   def size(Tree(val,left,right)) :
       1 + left.size() + right.size()

But that's not quite going to work. If this really were a more statically typed language like Haskell, and we knew that left and right were always Trees, this would be OK. But as we're trying to go for Pythonic duck-typing we have a problem. We would still like left.size() to mean, "pass the method selector size() to the object left" and then let the class of left decide how to handle it. But where left is None we have no idea which class to use to call. We could have a multitude of classes which define size(None); which would we pick?

Note that this is not the pattern-matching's fault. We'd have the same problem here :

 class Tree(val,left,right) :
    def size(self) : 
      if self is None : 0
      else : 
        1 + self.left.size() + self.right.size()
         

We could get around the problem by disallowing methods which take a single None argument, effectively forcing the programmer to write something like this :

 class Tree(val,left,right) :
   def size(Tree(_,None,None)) : 0
   def size(Tree(_,None,right)) : 1+right.size()
   def size(Tree(_,left,None)) : 1+left.size()
   def size(Tree(val,left,right)) :  
       1 + left.size() + right.size()

But this starts looking somewhat clunky.

We might make it a bit less awkward by providing conditional evaluation to the left and right subtrees. Based on Python's current conditional expression syntax that would look something like this :

 class Tree(val,left,right) :
    def size(Tree(_,left,right)) : 
       1 + (left.size() if left else 0) +  
           (right.size() if right else 0)

Or maybe we have to accept a static type system closer to Haskell's. Perhaps the types, in this case, contribute to Haskell's terseness.

Kira is a language which compiles to PHP