Monads in Python (Again)

Dustin Getz provides one of the best "Monads for idiot Python programmers" explanations I've seen.

Excellent! I think I almost do understand this one.

GrabQuora on GitHub

A couple of tweaks to the Quora grabbing script. Makes it worth upgrading to a full GitHub project.

Quora Scraper

I love Quora. It's a great site and community. But I started getting a bit concerned how much writing I was doing there which was (potentially) disappearing inside their garden and not part of the body of thinking I'm building up on ThoughtStorms (or even my blogs).

Fortunately, I discovered Quora has an RSS feed of my answers, so I can save them to my local machine. (At some point I'll think about how to integrate them into ThoughtStorms; should I just make a page for each one?)

Anyway here's the script (powered by a lot of Python batteries.)




And this turns the files back into a handy HTML page.

Java Hater

Someone on Quora asked me to answer a question on my personal history of using Java. It became a kind of autobiographical confession. 

I've never had a good relationship with Java.

My first OO experience was with Smalltalk. And that spoiled me for the whole C++ / Java family of strongly typed, compiled OO languages. 

Because I'd learned Smalltalk and this new fangled OO thing when it was still relatively new (in the sense of the late 80s!) I thought I had it sussed. But actually I had very little clue. I enthusiastically grabbed the first C++ compiler I could get my hands on and proceeded to spend 10 years writing dreadful programs in C++ and then Java. I had assumed that the OOness of both these languages made them as flexible as I remembered Smalltalk to be. I thought that OO was the reason for Smalltalk's elegance and that C++ and Java automatically had the same magic. 

Instead I created bloated frameworks of dozens of classes (down to ones handling tiny data fragments that would have been much better as structs or arrays). I wrote hugely brittle inheritance hierarchies. And then would spend 3 months having to rewrite half my classes, just to be able to pass another argument through a chain of proxies, or because somewhere in the depths of objects nested inside objects inside objects I found I needed a new argument to a constructor. The problem was, I was programming for scientific research and in industry but I hadn't really been taught how to do this stuff in C++ or Java. I had no knowledge of the emerging Pattern movement. Terms like "dependency injection" probably hadn't even been invented. 

I was very frustrated. And the funny thing I started to notice was that when I had to write in other languages : Perl, Javascript, Visual Basic (Classic), even C, I made progress much faster. Without trying to model everything in class hierarchies I found I just got on and got the job done. Everything flowed much faster and more smoothly.

Perl's objects looked like the ugliest kludge, and yet I used them happily on occasion. In small simulations C structs did most of what I wanted objects to do for me (and I did finally get my head around malloc, though I never really wrote big C programs). And I had no idea what the hell was going on with Javascript arrays, but I wrote some interesting, very dynamic, cross browser games in js (this is 1999) using a bunch of ideas I'd seen in Smalltalk years before (MVC, a scheduler, observer patterns etc.) and it just came out beautifully. 

It wasn't until the 2000s that I started to find and read a lot of discussions online about programming languages, their features, strength and weaknesses. And so I began my real education as a programmer. Before this, a lot of the concepts like static and dynamic typing were vague to me. I mean, I knew that some languages you had to declare variables with a type and in some you didn't. But it never really occurred to me that this actually made a big difference to what it was like to USE a language. I just thought that it was a quirk of dialect and that good programmers took these things in their stride. I assumed that OO was a kind of step-change up from mere procedural languages, but the important point was the ability to define classes and make multiple instances of them. Polymorphism was a very hazy term. I had no real intuitions about how it related to types or how to use it to keep a design flexible.

Then, in 2002 I had a play with Python. And that turned my world upside-down.
For the first time, I fell in love with a programming language. (Or maybe the first time since Smalltalk, which was more of a crush).

Python made everything explicit. Suddenly it was clear what things like static vs. dynamic typing meant. That they were deep, crucial differences. With consequences. That the paraphernalia of OO were less important than all the other stuff. That the fussy bureaucracy of Java, the one class per file, the qualified names, the boilerplate, was not an inevitable price you had to pay to write serious code, but a horribly unnecessary burden.
Most of all, Python revealed to me the contingency of Java. In the small startup where I'd been working, I had argued vehemently against rewriting our working TCL code-base in Java just because Java was OO and TCL wasn't. I thought this was a waste of our time and unnecessary extra work. I'd lost the argument, the rewrite had taken place, and I hated now having to do web-stuff with Java. Nevertheless, I still accepted the principle that Java was the official, "grown up" way to do this stuff. Of course you needed proper OO architecture to scale to larger services, to "the enterprise". Ultimately the flexibility and convenience of mere "scripting" languages would have to be sacrificed in favour of discipline. (I just didn't think we or our clients needed that kind of scaling yet.) 

What Python showed me was we weren't obliged to choose. That you could have "proper" OO, elegant, easy to read code, classes, namespaces, etc. which let you manage larger frameworks in a disciplined manner and yet have it in a language that was light-weight enough that you could write a three line program if that's what you needed. Where you didn't need an explicit compile phase. Or static typing. Or verbosity. Or qualified names. Or checked exceptions. What I realised was that Java was not the inevitable way to do things, but full of design decisions that were about disciplining rather than empowering the programmer. 

And I couldn't stomach it further. Within a few months of discovering Python I had quit my job. Every time I opened my machine and tried to look at a page of Java I felt literally nauseous. I couldn't stand the difference between the power and excitement I felt writing my personal Python projects, and the frustration and stupidity I felt trying to make progress in Java. My tolerance for all Java's irritations fell to zero. Failing to concentrate I would make hundreds of stupid errors : incompatible types, missing declarations or imports, forgetting the right arguments to send to library methods. Every time I had to recompile I would get bored and start surfing the web. My ability to move forward ground to a halt.
I was so fucking happy the day I finally stopped being a Java programmer.

Postscript : 

1) Something I realized a while after my bad experience was how important the tools are. My period in Java hell was trying to write with Emacs on a small-screen laptop without any special Java tools (except basic Java syntax colouring). I realize this is far from the ideal condition to write Java and that those who are used to Eclipse or IntelliJ have a totally different experience and understanding of the language. 

2) A few years later, I taught the OO course in the local university computer science department. All in Java. By that time, I'd read a couple of Pattern books. Read Kent Beck's eXtreme Programming. Picked up some UML. And I had a much better idea what Polymorphism really means, how to use Interfaces to keep designs flexible, and why composition is better than inheritance. I tried to get the students to do a fair amount of thinking about and practising refactoring code, doing test driven development etc. It all seemed quite civilized, but I'm still happy I'm not writing Java every day. 

3) A couple of years ago I did do quite a lot of Processing. I was very impressed how the people behind it managed to take a lot of the pain of Java away from novice programmers. I wonder how far their approach could be taken for other domains.

Personal Question

Question : Hey Phil, do you actually do any programming these days?

Answer : Yes. Quite a lot at the moment. Though it's a bit all over the shop.

I'm dipping a toe into Android programming. (And, hmmm ... Java .... I thought I'd got over my Java hangups by doing a lot of Processing, but it turns out that Processing just hides the crap and Android doesn't. Why hasn't Google picked up on Processing to turn it into a first-class Android art / game app. development environment?)

I'm mainly writing CoffeeScript. Some stuff related to my ongoing 3D modelling / desktop manufacturing projects. (Did I forget to mention those? I'm sure there's a half-written blogpost somewhere.) Some work towards an SdiDesk-derived network diagramming plugin for Smallest Federated Wiki (held up by silly problems). Some other bits and pieces. I've recently been playing with Jison, which rocks. And I'm about to investigate angular.js which looks pretty good.

There's a project for small stand-alone web-servers that I'll talk about more if / when it takes off.

I've been trying to compile example VST instruments  (C++) for some of my work with the Brasilia Laptop Orchestra, but it's driving me crazy. (I may go back to Pure Data which can be embedded in a VST.)

A bit of PHP, just simple small web-services.

I'm going to be teaching an Arduino course soon. So I'll be writing a bit of C and I want to try Occam-Ⲡ.

I'm still writing Python too. Mainly for short file transformation scripts or to prototype algorithms that later get translated into CoffeeScript.

Some of this stuff is headed for GitHub soon.

Command 'n' Cursor

I've been travelling with my trusty (but ageing) eeepc netbook this last week. There's much to love about it but it's starting to feel slow in comparison with my other machine.

Increasingly when I use the netbook I try to get away with doing things in a ctrl-alt-f1 shell without logging in to the GUI at all. I'm starting to wish more software could be used in this environment so I began to look at Curses, the standard library for text-window UIs. There's a convenient Python wrapper of course. And there's another nice library in Python : Cmd, for creating a command-line driven apps. That is, not programs that literally run as small tools on the shell with command-line arguments, but programs which have their own internal "repl" style loop which you drive by typing in commands. Cmd handily hides the details from you, letting you declare a subclass of the Cmd class which simply defines handlers for specific commands. It's not a million miles away from something I ended up writing to handle the commands in SdiDesk.

For some of my projects it would be useful to combine the two modes : to have Cmd style input driving a 2D textual output using Curses. Unfortunately Cmd and Curses don't obviously play well together.  Both of them want to take over the input, with Curses thinking in terms of keystrokes while Cmd still expects full lines.

Nevertheless, after a bit of exploration, and learning about Curses's textpads and Cmd's supplementary methods, I'm starting to get the two to co-operate. As this gist shows :



It doesn't do anything yet. Just handles a "greet NAME" command that prints "hello NAME". And a "quit" command that exits the program. But it has combined Cmd inputs with Curses output.

Permutations with Python Generators

Here's something neat.

I wanted to experiment creating different permutations of a collection of items. (In fact I'm working on some code for laying out shapes on a surface.)

Prototyping in Python to get my ideas straight I came up with this neat generator solution.

CoffeeScript

Just a note. I am really, really liking CoffeeScript now.

It's reminding me both of freedom that Python gave me when I first turned to it after Java. And bit of my experience with Erlang. ( If only it had Erlang's Actor model and pattern matching arguments ... )

The other good effect of this, CoffeeScript is making me more comfortable with investing my time writing serious logic on the browser-side. Which is where it should be, given the requirements of modern applications and that the browser is becoming the default GUI. 

Verpa's Gmail lib

Playing with this library to access my Gmail account via IMAP. Seems pretty simple and convenient, though fairly basic.

Been browsing some interesting discussion over at Quora on how they built their site.

It's basically Python and Pylons. But this is cool. They don't use a templating language.

Here's developer Charlie Cheever :

What "templating" means to most people is a way of having the developer write out HTML basically the way that you would send it to the browser and then having a way to include a few things -- typically variable substitution by using special tags lik <% ... %> or similar.

In our case, no one writes any code that looks like HTML/XML literals, so there's nothing in our codebase that really matches what most people think of as templates. We do have view code but that interleaves calls into the model and application logic along with a Python code description of what the HTML for that component should be, which is different from templates which are usually based around the ideas of separating logic and data fetching from this.

This sounds like an approach I've been favouring for a while. I did it in Mind Traffic Control, some other unreleased SdiDesk in Python experiments, and I do it in some Javascript I've written. People think that you should separate HTML from code because HTML is the domain of designers and code is for programmers. But I think HTML is the realm of data-structure (designers should stick to CSS) and part of the programmers' remit.

The way a programmer (or at least, this programmer) wants to express complex data structures is with function composition. So here's an example of my html.py file.

# HTML library
# basic level 

def tag(name,x,*argv) :
 if x is None :
  return u"<"+name+u"/>"
 if argv != (None,) : 
  inside = u''.join(argv)
 else :
  inside = u''
 if isinstance(x,dict) :
  # we're passing a dictionary of attributes for the tag
  s = u"<%s " % name 
  s = s + ' '.join(['%s="%s"'%(k,v) for (k,v) in x.iteritems()])
  s = s + u">"+inside+u"</"+name+u">"
  return s

 # or there are no attributes, just inner
 return u"<"+name+u">"+x+inside+u"</"+name+u">"


# Now we'll actually make some tags
tags = ['html','head','body','script','p','div','table',
        'tr','th','td','ul','ol','li','dt','dd','h1','h2',
        'h3','h4','h5','h6', 'style','pre']

loc = locals()
def setit(loc,t) :
 loc[t] = lambda x=None,*argv : tag(t,x,*argv)

for t in tags :
 setit(loc,t)


# Use like this 
html(
    head(),
    body(
        h2("Header"),
        p('para1'),
        p('para2')
    )
)

But I did start to wonder, given the prevalence of templating languages and some of my recent experiences as a Django developer, whether this wasn't just me being wilfully perverse / crazy. I admit I'm kind of relieved to read that Quora are doing something similar. Maybe I wasn't so mad after all.

Bonus link : Decomposition by language is probably a modularity mistake. (Written back when I was more confident.)

Resolver Systems have a new "cloud-based" pythonic spreadsheet called "Project Dirigible"

This is an absolutely brilliant summary of the virtues of PHP.

The important point is that these virtues aren't going away. By comparison this seems to miss the point. In 2020 we won't be programming the web with an advanced Python framework (wonderful as python is). We'll have something which does what PHP did for CGI or Processing does Java, ie. wrap a purpose built, sophisticated back-end (something like Google Application Engine) in a light, domain-specific language. That language won't look like PHP. It would be nice if it looked like Python, but I suspect Javascript is a more likely model.

But it will retain the virtues of PHP : none of this fussy separation of presentation and logic; easy discoverability of where URLs go; fast iterative development; big built-in library etc.


(Hat-tip, BillSeitz for the links)

A quick thought ... working with a lot of Django and Pinax at the moment, I'm seeing a lot of use of Python's varargs. Functions that just take *argv, **kwargs

I find I don't like this. I want to see argument lists as explicit as possible. I feel uncomfortable and a bit lost. I feel unprotected when the compiler can't even check the number of arguments I'm sending.

So I wonder if this is legitimate. Or isn't it analogous to the case of static typing? Varargs give more flexibility than fixed arguments. Much as dynamic typing is more flexible than static. So why shouldn't I prefer the power and flexibility over restriction and security in this case?

Update : Of course, this is the way that GeekWeaver works, where there's no explicit definition of the list of arguments passed to a reusable block. I've been unhappy with that there too. Thinking that I should add positional explicit named arguement lists. But if this kwargs thing is a trend, then GW should probably be left as it is.

Playing with simple python access to RabbitMQ today. Looks pretty cool.

Oh God!

Beards of Python

Compare and contrast

This is possibly the best StackOverflow thread ever (on the quirks of Python closures)

Python 3000 released

I didn't know some of these great Python features.

PythonCard, the Hypercard / VB-like python development environment maybe being forked and revivified.

Follow on from yesterday's "Python / Haskell crossbreed" post. Both Al "Folknology" and Gleber point me at the Reia programming language. A Python / Ruby like scripting language on top of the Erlang Open Telecom Platform (Erlang's parallel virtual machine).

Very sweet ... I've subscribed to the mailing list to find out more.