Steven Pinker at Congress Centre

Great talk taken from his new book The Stuff of Thought. I won’t cover the sophisticated and interesting arguments here (read the book!) but here are some nuggets on use of language …

Box of crayons that looks like an audience because it is in rows and tiered

Child to her father:

I don’t want the flat crayons I want the ones that look like an audience.

Groucho Marx classics:

I’ve had a great evening. But this wasn’t it.

If I held you any closer I’d be on the other side of you.

Why does a man with hair on his head have more hair than a man with hairs on his head?

Was the World Trade Centre attack on September 11th 2001 one incident or two? There was a 3.5 billion dollar lawsuit based on the interpretation of a maximum payout of $3.5 billion dollars “per incident”.

One of the reasons detectives suspected Scott Peterson of the murder of his wife was that he referred to her in the past tense before her body was found.

James Garfield‘s assassin’s remark following the death of the President of the United States from complications arising from poor medical care after being shot three months earlier:

The doctors killed him. I just shot him.

The Federal Communications Comission chose not to censure Bono over his remark that the award he was receiving was “really, really fucking brilliant” as it did not refer to a sexual act but was rather used to emphasize. Further legislation was passed as a result of this. Although the legislation goes into graphic detail over what words are not allowed it fails to make illegal the use of fucking as an emphasizing adverb.

That “damn tabernacle” and “damn chalice” are swearing in Quebecois French.

The description of a certain mode of speech as “fuck patois”. And an illustrative tirade from a soldier returning from war:

You fucking mean I’ve fucking come all the way fucking home from the fucking war to my fucking house to find my fucking wife having illicit sexual relations with my fucking neighbour?

The editor of Gourmet magazine challenged one of his journalists to bribe the maître d’s of New York’s finest restaurants $50 to seat him and his partner without reservations. It worked each and every time.

Simple Haskell Web Programming with HAppS

Motivation

There are two or three CGI libraries for Haskell. But if you ask the cognoscenti (to be found on the #haskell IRC channel) for Haskell web programming recommendations they will almost invariably point you at HAppS.

HAppS may have many virtues but it’s documentation and associated tutorial do not number among them.

Copying the first Rails screencast, Creating a Blog in Fifteen Minutes, I figured that a good example web application is a blogging system. Leaving aside that blogging is now pass�, here is my HAppS tutorial.

Please note that I am a fairly inexperienced Haskell programmer and almost all the advice I give will be wrong. Some of what I say will necessarily be vague because I don’t understand it completely myself. But the advantage to my tutorial is that it will at least be comprehensible! I hope that some Haskell gurus will read this and find my mistakes and update me in the comments. Thanks!

Installation

For installation of HAppS, please see my earlier post.

Hello, World

First, let’s get HAppS working and see something in our browser. Save the following listing to Hello.hs:

import HAppS
import HAppS.Protocols.SimpleHTTP2
 
main = stdHTTP [debugFilter, noState, h () GET $ ok $ val "Hello, World"]

If you compile that with:

ghc --make Hello.hs -o hello

And then run the resulting “hello” binary (./hello under Unix, hello under Windows). If you visit http://localhost:8000/ in your browser you should see “Hello, World” displayed.

So what have we actually done there? Well, I don’t really know. But certainly we’ve passed a list of directives to the stdHTTP function. One of which (the bit starting “h”) says: “if you get a GET request of any sort, respond with a page containing the text ‘Hello, World'”. Let’s just leave it at that for now.

Blog System

Weren’t we meant to building a blogging system? Yes! OK, let’s start with the URL scheme. We want the following paths in our system:

GET  /new 		=> Show the "create new post" form
POST /new	 	=> Create a new post using the submitted values
GET /post/$id 	=> Display the post with id $id
GET /			=> Dispaly the last 5 posts

The way to describe this scheme to HAppS is through the first and second arguments to the h request-handling function. The first argument is the path as a list (each element in the list representing a directory) and the second argument is the type of request (GET, POST and so on). So for displaying the “create new post form” we use h ["new"] GET.

Our GET /post/$id path is slightly tricky in that we don’t want to write out a very long list of post ids. We want to do some pattern matching. You can do this in HAppS with the re function. This tells HAppS to interpret the paths in the list as regular expressions. So for GET /post/$id we can use h ["post", "(0-9 )"] GET.

Putting that all together gives us:

import HAppS
import HAppS.Protocols.SimpleHTTP2
 
main = stdHTTP $ debugFilter : noState : 
           [h ["new"]                           GET  $ ok $ val "Show create post form"
           ,h ["new"]                           POST $ ok $ val "Save post and forward to display post"
           ,h (re ["post", "([0-9] )"])  GET  $ ok $ \\(id:_) () -> respond ("Display post "    id)
           ,h [""]                              GET  $ ok $ val "Display five most recent posts"
           ]

HTML in Haskell

Those calls to val are just placeholders. What we really want to return is some HTML.

The most common way to get HTML in Haskell seems to be to use one of the HTML combinator libraries such as Text.XHTL.Strict. This is only tangentially related to HAppS so I’ll just point you at this simple example and these fairly comprehensible notes and move on.

Now we have:

import HAppS
import HAppS.Protocols.SimpleHTTP2
import Text.XHtml.Strict

main :: IO ()
main = stdHTTP $ debugFilter : noState : 
           [h ["new"]                           GET  $ ok $ val showCreatePostPage
           ,h ["new"]                           POST $ ok $ val "Save post and forward to display post"
           ,h (re ["post", "([0-9] )"])  GET  $ ok $ \\(id:_) () -> respond ("Display post "    id)
           ,h [""]                              GET  $ ok $ val "Display five most recent posts"
           ]

-- Handlers

showCreatePostPage :: String
showCreatePostPage = showHtml getCreatePostPage

-- HTML Functions

getCreatePostPage :: Html
getCreatePostPage = mkHeader "New Post"
        body < < [
        (h1 << "New Post") 
            getCreatePostForm
        ]

getCreatePostForm :: Html
getCreatePostForm = (form ! [action "/new", method "post"] << (fieldset << [
        label ! [thefor "title"] << "Post Title", br,
        input ! [name "title", identifier "title", thetype "text"], br, br,
        label ! [thefor "body"] << "Post Body", br,
        textarea ! [name "body", identifier "body", rows "10", cols "40"] << "", br, br,
        submit "create" "Create Post"
    ]))

mkHeader :: String -> Html
mkHeader title = (header < < (thetitle << (title    " - HBlog")))

State

Of course I've chickened out above. Yes, I've provided some HTML that never changes. But what about the pages that depend on state? What about the display of the posts themselves? It's time to get rid of that odd-looking noState directive and start storing some blog posts. This is where it gets a little bit hairy.

While you could use a relational database with HAppS, it actually has it's own in-built haskell type-based storage system. It has lots of whizzy features that I'm not concerned about here to do with atomicicity and all that kind of shenanigans. But let's stay focused - we need a way of storing some blog posts. That means we need to define a type that derives Show and Read.

data Posts = Posts [Post] deriving (Read, Show) 
data Post = Post { pTitle :: String, pBody :: String } deriving (Read, Show)

What’s more we have to give it some instances so that HAppS can start it up, and know how to store it.

instance Serialize Posts where 
    typeString _ = "Posts"
    encodeStringM = defaultEncodeStringM
    decodeStringM = defaultDecodeStringM

instance StartState Posts where
    startStateM = return $ Posts []

OK, so now we know where we are going to store our posts. But how do we construct them in the first place? Well, Post is a record type, I will assume you know how to use record types. But what about extracting the values from the post body to slot into the record?

Strings we can get with:

v < - lookMb return req "fieldName"

where req is the Request, the second parameter passed to the function after ok in our request handlers. This code must appear in a do block.

Ints we can get with:

v < - lookMb readM req "fieldName"

Quite what the meaning of this distinction is, I don't know. So please leave a comment if you understand this better than me.

The state of the application itself (in our case a Posts, or list of Post) you can retrieve with get and update with put as long as you are within the appropriate Monad. This probably makes more sense in the context of a full example:

module Main where

import Control.Monad.State (get, put)
import Debug.Trace
import HAppS
import HAppS.Protocols.SimpleHTTP2
import Text.XHtml.Strict

main :: IO ()
main = stdHTTP $ debugFilter : 
           [h ["new"]                           GET  $ ok $ val showCreatePostPage
           ,h ["new"]                           POST $ ok createPost
           ,h (re ["post", "([0-9] )"])         GET  $ ok showPost
           ,h [""]                              GET  $ ok showPosts
           ]

-- State

data Posts = Posts [Post] deriving (Read, Show) 
data Post = Post { pTitle :: String, pBody :: String } deriving (Read, Show)

instance Serialize Posts where 
    typeString _ = "Posts"
    encodeStringM = defaultEncodeStringM
    decodeStringM = defaultDecodeStringM

instance StartState Posts where
    startStateM = return $ Posts []

-- Handlers

showCreatePostPage = showHtml getCreatePostPage

showPost (rawId:_) () = do
    (Posts posts) < - get
    let postId = read rawId :: Int
        post = posts !! postId 
    respond $ showHtml $ getPostsHtml [post]

showPosts () () = do
    (Posts posts) <- get
    respond $ showHtml $ getPostsHtml $ take 5 $ reverse posts 

createPost () req = do
    title <- lookMb return req "title"
    body  <- lookMb return req "body"
    let p = Post { pTitle = title, pBody = body }
    (Posts ps) <- get
    put (Posts (ps    [p]))
    respond $ "Post created"

-- HTML Functions

getCreatePostPage :: Html
getCreatePostPage = mkHeader "New Post"
        body << [
        (h1 << "New Post") 
            getCreatePostForm
        ]

getCreatePostForm :: Html
getCreatePostForm = (form ! [action "/new", method "post"] << (fieldset << [
        label ! [thefor "title"] << "Post Title", br,
        textfield "title", br, br,
        label ! [thefor "body"] << "Post Body", br,
        textarea ! [name "body", identifier "body", rows "10", cols "40"] << "", br, br,
        submit "create" "Create Post"
    ]))

getPostsHtml :: [Post] -> Html
getPostsHtml posts = foldl (   ) noHtml (map getPostHtml posts)

getPostHtml :: Post -> Html
getPostHtml post = 
    thediv 
        < < h2 << (pTitle post)
            p << (pBody post)

mkHeader :: String -> Html
mkHeader title = (header < < (thetitle << (title    " - HBlog")))

Once your HAppS application has state then wherever you run it it will create directories to store state information. If you change your state type, you must delete these directories (you will lose existing state of your app) to prevent errors like: *** Exception: user error (decodeStringM: parsing length field failed @ "").

Conclusion

So there you have a working blog publishing system in HAppS. It's more a proof of concept than a useable application. But hopefully it shows you how to create a simple application in HAppS without boiling your brain too much. There are zillions of improvements that could be made: add comments; make Posts a sequence and not a list; add error handling for empty fields; redirect to post display after creation; handle error when post id is greater than the number of posts; don't number posts starting from 0; allow deletion of posts; and many many others.

I must credit Eric Mertens for his sterling work on HPaste whose source code I read to make sense of much of what HAppS was doing.

Please do correct all my terrible misunderstandings and mistakes in the comments!

How to Install HAppS

Update: these instructions refer to the HAppS, 0.8.8. The latest version (September 2007) if 0.9.1a. It radically alters the HAppS APIs as well as the installation procedure. You can find more information here which is an incomplete tutorial inside one of the new source code repositories.

  1. Get the latest version of ghc for your platform:

    http://haskell.org/ghc/

  2. Get the latest version of darcs:

    http://wiki.darcs.net/DarcsWiki/CategoryBinaries

  3. Throughout, when building HAppS or any of it’s dependencies use:

    $ runhaskell Setup.hs configure
    $ runhaskell Setup.hs build
    $ sudo runhaskell Setup.hs install
    

    and not the makefiles. Even where the README or web page says otherwise. In some instances Setup.lhs will be provided instead of Setup.hs.

  4. Use the latest source for HAppS 0.8.8 not the stable version. The stable version won’t build! Get it via darcs not tarballs. The tarballs are broken links!

    darcs get --partial http://happs.org/HAppS
    
  5. Try and build HAppS (with the commands in step 3 above), and every time you fail, install the missing packages. The documentation in the HAppS tutorial is up-to-date (as of September 2007) on this score. So you may be better off there than here if it is long after that date.

    The hackage package list is useful but not complete.

    You may need: HaXML, binary, hslogger.

    HList is a special case as it doesn’t have a conveniently downloadable package. Use:

    wget -r -nH -X '*/_darcs' -np http://darcs.haskell.org/HList/
    

    Plus the following are only available via darcs: syb-with-class ( darcs get http://happs.org/HAppS/syb-with-class ), default ( darcs get http://happs.org/HAppS/default ), normalize ( darcs get http://happs.org/HAppS/normalize ), generic-xml ( darcs get http://happs.org/HAppS/generic-xml )

You can test your installation with the “Hello World” in the HAppS Tutorial

Update: the rest of the HAppS tutorial was fairly dense and difficult to understand as well as being somewhat out of date. To remedy this situation I wrote Simple Haskell Web Programming with HAppS.

Prevent Browser Caching

You can’t force a browser (or other User Agent) to do anything. You must carefully implement your server side code to prevent malicious or accidental damage. That said, you can sometimes improve the user experience a lot by asking browsers nicely not to cache anything and thus to request the page again when (amongst other things) the Back button is used.

You can do this with the following HTTP headers. (Note: This doesn’t work on Opera as it does on other browsers.)

Cache-Control: no-cache
Cache-Control: no-store
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Pragma: no-cache

In PHP you can do this with:

header("Cache-Control: no-cache");  // Forces caches to obtain a new copy of the page from the origin server
header("Cache-Control: no-store");  // Directs caches not to store the page under any circumstance
header("Expires: " . date('D, d M Y H:i:s', 0) . ' GMT'); //Causes the proxy cache to see the page as "stale"
header("Pragma: no-cache");         // HTTP 1.0 backward compatibility

This is not the perfect solution. If the browser ignores these headers (as Opera will – see comments) then you can still go back and see stale pages. I wonder what banks do to get around this where the viewing of a stale page can be considered a security breach?

Combinatorics in Ruby

Some simple additions to Integer for combinatorics purposes. Will calculate the number of possibilities in an “n choose m” scenario (no repetitions). Adds factorial because that is required for the calculation.

Usage:

irb(main):001:0> require 'combinatorics'
=> true
irb(main):002:0> 3.fact
=> 6
irb(main):003:0> 10.fact
=> 3628800
irb(main):004:0> 100.fact
=> 93326215443944152681699238856266700490715968264381621468592963895217599993229915608941463976156518286253697920827223758251185210916864000000000000000000000000
irb(main):005:0> 3.choose(2)
=> 3
irb(main):006:0> 6.choose(2)
=> 15
irb(main):007:0> 100.choose(7)
=> 16007560800
irb(main):008:0> 10000.choose(27)
=> 88666586562493644592361321224866562980818106140387273533229708768701222758880000

Code:

class Integer
  def choose(m)
    return (self.fact / (m.fact * (self - m).fact))
  end

  def fact
    (2..self).inject(1) { |f, n| f * n }
  end
end

Science in Virtual Worlds

I attended the Science in Virtual Worlds event at the London Apple store yesterday evening. I’ve always been quite cynical about systems like Second Life but I am having second thoughts now.

SciLands is an interesting place. The National Physical Laboratory and NASA (amongst others) have created an area dedicated to hard science and science education.

Dave Taylor of the NPL made a spirited case for the usefulness of Second Life. In addition to bringing geographically divided people together (much like email or instant messaging can) he argued that the use of a three-dimensional world had extra advantages.

It increases the sociability of situations (with real world cues such as stepping towards the centre to speak to a group arranged in a circle). This makes the interactions richer and more “real” than something like email. It also encourages informal conversations. For example, after a video conference the camera is switched off but after a Second Life conference, avatars must move away or can form smaller groups, renew acquaintances and so on as in a real conference. Serendipity is also increased by allowing anyone anywhere on the globe to be neighbours. One of NPL’s areas borders the University of Denver and they are now engaged on a joint project to bring a nuclear power station to Second Life, something that would not have happened were they not neighbours.

Rsync Exit Codes

  • 0 Success
  • 1 Syntax or usage error
  • 2 Protocol incompatibility
  • 3 Errors selecting input/output files, dirs
  • 4 Requested action not supported: an attempt was made to manipulate 64-bit files on a platform that cannot support them; or an option was specified that is supported by the client and not by the server.
  • 5 Error starting client-server protocol
  • 10 Error in socket I/O
  • 11 Error in file I/O
  • 12 Error in rsync protocol data stream
  • 13 Errors with program diagnostics
  • 14 Error in IPC code
  • 20 Received SIGUSR1 or SIGINT
  • 21 Some error returned by waitpid()
  • 22 Error allocating core memory buffers
  • 23 Partial transfer due to error
  • 24 Partial transfer due to vanished source files
  • 30 Timeout in data send/receive

Rails Default Controller for Site Root

To designate a particular controller so that all of its methods appear off the root of your site, add the following to routes.rb inside the ActionController::Routing::Routes.draw do |map| block:

    map.connect ':action', :controller => "yourcontrollername"

Web App in 30 Hours

It’s a bank holiday weekend. What better way to spend it than to try to write a web application in 30 hours?!

I’m going to write an application to search eBay, Amazon, free ebooks, TV guides, cinema listings and down the back of the sofa for books, movies and music. It will offer persistent search so you can say, “I want a copy of Catcher in the Rye but I don’t want to pay more than �3 for it.” It will offer to email you results, or create an RSS feed.

What I’m definitely going to need is a good eBay library and a good Amazon library. Ruby (a language I really like) seems to have both of those. Python has easybay, and some kind of Amazon library but neither seem quite as mature as the Ruby equivalents. I’d like to use web.py with Python as I’ve never really got on with Rails like some people seem to. But perhaps this can be the app that changes all that. So for Hour One at least I’m going to be writing this in Ruby using Rails.

In the spirit of doing “the simplest thing that could possibly work” the initial app is going to be single text input page that searches eBay UK only and displays the results. Under source control, with unit tests, because I don’t want this to come crashing down around my ears 20 hours in.

OK, it’s already 0904 – I’m four minutes behind schedule &#8211 let’s go!

Hour One

We are off! Here’s what I managed in Hour One:

  • Created a finder skeleton app.
  • Installed darcs for source control and created a repository.
  • Spent too long trying to follow the instructions from Rails Recipes to get Rails running without a database, without success.
  • Created the Item and Ebay classes with some very basic unit tests.

Now to hook up to the actual eBay library and get that working.

Hour Two

I’ve got the remarkably nice Ruby eBay API client up and running. Now I need to get it to return results to my application.

In more detail:

  • Installed subversion, retrieved Ruby eBay API library and then realised that’s for hacking on it not using it.
  • Installed Ruby eBay API client gem.
  • Set up developer keys for developer.ebay.com
  • Created config file for Ruby eBay API client.
  • Had some breakfast.
  • Generated auth token for eBay sandbox user.
  • Created an eBay sandbox user.
  • Created test app that gets the official eBay time.
  • Created test listings through eBay lib.
  • Searched test listings with test app.

Hour Three

I’ve got the eBay client returning results into the application. I need to spend some more time getting attributes like current price and postage cost so that these results are more useful.

I also need to create the web front end for display of the search form and the search results.

  • Unit tests for ebay searcher.
  • Got ebay searcher returning FinderItems.
  • Created another eBay user to bid on my test auctions.
  • Had a good poke around the eBay API to see what attributes are available.

Hour Four

I found current price and URL in the eBay API so the FinderItems produced now include those details. I’ve started work on the web side of things. I have a basic search form and basic search results page. Now I just need to get the results page to display the FinderItems I’m passing to it in the controller.

  • Identified basic properties in eBay API and added them to FinderItem objects.
  • Installed irb for interactive ruby.
  • Generated basic rails controller for the app.
  • Created basic search form and basic search results page.
  • Created layout for header/footer for evey page in the app.
  • Started to hook EbaySearcher and FinderItem together with the web view/controller.

Hour Five

Quite a frustrating hour but things are moving on nonetheless. I have eBay search results from the sandbox display in the browser. I’m adding Amazon in before I solidify final design decisions like what attributes FinderItem is going to require so I don’t get caught out reproducing the eBay API with not enough flexibility to cope with other sources of data.

Loads of little tasks are starting to queue up — everything from making the search results into clickable URLs to adding a “please wait” screen. Plus I need to take some time out and do an actual design for the site.

  • Spent a little while chasing down a bug – rails doesn’t like you using include in the top-level namespace – you need to do it inside a class else you get “NameError: cannot remove Object”.
  • Got eBay results displaying in browser
  • Tried to move query => index and ended up deadlocking the webserver and moving it back.
  • Installed Ruby/Amazon library. Created AmazonSearcher with unit tests.

Hour Six

I’ve got Amazon and eBay results coming through in the web app now. There’s some serious issues with the eBay side of things (tries to retrieve more than 10,000 items from the API if you enter “test” as the search term) but looking good. I’ve also hidden the two existing data sources behind a Searcher class that keeps knowledge of exactly what’s going on away from things that shouldn’t know about it. Didn’t get much else done this hour – was too busy eating lasagne. Going to take an break now and start again in an hour.

  • Fixed up AmazonSearcher so it passes its unit tests.
  • Set search to redirect to the search form if no query is supplied.
  • Created Searcher which holds all searchers and amalgamates results.
  • Had lunch.

Hour Seven

Added a stylesheet so I can start to make it look like it should. Added in the concept of a User who may or may not be registered. Plus Location (get from IP initially, can be changed by the user). Everything is a bit haywire at the moment as I assimilate these changes.

Hour Eight

Had to take some time out to do some other things but now have a styled-up, eBay and Amazon searching app with a concept of User and Location. Upon initial visit you are silently logged in as a guest with a location guessed from your IP address. Eventually only sources that are relevant to your location will be searched.

I need to build the functionality to change your location and to register with the site. I also need to build in the concept of location to the existing eBay and Amazon searching libraries.

Still loads to do but lumbering towards a kind of “prototype” state.

Hours Nine and Ten

Last night I got the application running off the root of the site (localhost/ instead of localhost/finder/) and solidified a few architectural decisions.

This morning I’ve followed through on the architectural decisions and written any missing unit tests. There’s still one not passing to do with looking up location from IP address.

  • Now works off site root, not /finder/blah
  • Architecture of data sources, locations, etc. now clearer.

Hours Eleven and Twelve

Fixed up a lot of little bits and made prices/currency work how it should. Want to get on to persisting searches soon but lots of little things to do first.

  • Fix failing unit test
  • Sort results by price, lowest first
  • Added concept of currency, extended Money from money module

Hours Thirteen and Fourteen

Made quite a lot of progress yesterday afternoon but then had to go out for a friend’s birthday. I’m not going to make 30 hours over the long weekend but it’ll be good to see how far I can get today. Not sure if I will try and make something live tonight or finish off the 30 hours another day – I’ll decide that later on.

  • Make change location page work properly.
  • Get Amazon and eBay searchers to respect user location and search the appropriate site.
  • User registration.

Hour Fifteen – Half Way

Made good progress for an hour but had to stop working on it to produce a website at very short notice for the Lady Margaret short film.

I’ve used up half my allotted hours and the app is fairly functional. The big thing now is to add the persistent part of persistent search – to let users save their searches and be notified when new items become available. I also want to expand it to more data sources like abebooks.com. I’m not sure when I’ll get to do the other 15 hours but I definitely want to get something live quite soon.

  • Added France, Germany, Japan, Ireland.
  • Fixed for different currencies in Amazon searcher.
  • Proper message if there are no results.
  • Handle SearchErrors from Amazon.

London Cinema Revamped

I’ve revamped londoncinema.bluebones.net.

As well as a new design it has much-improved cinema and film pages, movie posters, trailers and imdb.com ratings.

Search is now even better because as well as taking your location it also uses imdb.com ratings to push better films higher up. With so many films showing it’s so easy to miss something good and with so many cinemas in London it’s so much better than trawling through cinema-by-cinema.