bluebones.net

It’s a bank holiday weekend. What better way to spend it than to try to write a web application in 30 hours?!

I’m going to write an application to search eBay, Amazon, free ebooks, TV guides, cinema listings and down the back of the sofa for books, movies and music. It will offer persistent search so you can say, “I want a copy of Catcher in the Rye but I don’t want to pay more than �3 for it.” It will offer to email you results, or create an RSS feed.

What I’m definitely going to need is a good eBay library and a good Amazon library. Ruby (a language I really like) seems to have both of those. Python has easybay, and some kind of Amazon library but neither seem quite as mature as the Ruby equivalents. I’d like to use web.py with Python as I’ve never really got on with Rails like some people seem to. But perhaps this can be the app that changes all that. So for Hour One at least I’m going to be writing this in Ruby using Rails.

In the spirit of doing “the simplest thing that could possibly work” the initial app is going to be single text input page that searches eBay UK only and displays the results. Under source control, with unit tests, because I don’t want this to come crashing down around my ears 20 hours in.

OK, it’s already 0904 – I’m four minutes behind schedule &#8211 let’s go!

Hour One

We are off! Here’s what I managed in Hour One:

• Created a finder skeleton app.
• Installed darcs for source control and created a repository.
• Spent too long trying to follow the instructions from Rails Recipes to get Rails running without a database, without success.
• Created the Item and Ebay classes with some very basic unit tests.

Now to hook up to the actual eBay library and get that working.

Hour Two

I’ve got the remarkably nice Ruby eBay API client up and running. Now I need to get it to return results to my application.

In more detail:

• Installed subversion, retrieved Ruby eBay API library and then realised that’s for hacking on it not using it.
• Installed Ruby eBay API client gem.
• Set up developer keys for developer.ebay.com
• Created config file for Ruby eBay API client.
• Had some breakfast.
• Generated auth token for eBay sandbox user.
• Created an eBay sandbox user.
• Created test app that gets the official eBay time.
• Created test listings through eBay lib.
• Searched test listings with test app.

Hour Three

I’ve got the eBay client returning results into the application. I need to spend some more time getting attributes like current price and postage cost so that these results are more useful.

I also need to create the web front end for display of the search form and the search results.

• Unit tests for ebay searcher.
• Got ebay searcher returning FinderItems.
• Created another eBay user to bid on my test auctions.
• Had a good poke around the eBay API to see what attributes are available.

Hour Four

I found current price and URL in the eBay API so the FinderItems produced now include those details. I’ve started work on the web side of things. I have a basic search form and basic search results page. Now I just need to get the results page to display the FinderItems I’m passing to it in the controller.

• Identified basic properties in eBay API and added them to FinderItem objects.
• Installed irb for interactive ruby.
• Generated basic rails controller for the app.
• Created basic search form and basic search results page.
• Created layout for header/footer for evey page in the app.
• Started to hook EbaySearcher and FinderItem together with the web view/controller.

Hour Five

Quite a frustrating hour but things are moving on nonetheless. I have eBay search results from the sandbox display in the browser. I’m adding Amazon in before I solidify final design decisions like what attributes FinderItem is going to require so I don’t get caught out reproducing the eBay API with not enough flexibility to cope with other sources of data.

Loads of little tasks are starting to queue up — everything from making the search results into clickable URLs to adding a “please wait” screen. Plus I need to take some time out and do an actual design for the site.

• Spent a little while chasing down a bug – rails doesn’t like you using include in the top-level namespace – you need to do it inside a class else you get “NameError: cannot remove Object”.
• Got eBay results displaying in browser
• Tried to move query => index and ended up deadlocking the webserver and moving it back.
• Installed Ruby/Amazon library. Created AmazonSearcher with unit tests.

Hour Six

I’ve got Amazon and eBay results coming through in the web app now. There’s some serious issues with the eBay side of things (tries to retrieve more than 10,000 items from the API if you enter “test” as the search term) but looking good. I’ve also hidden the two existing data sources behind a Searcher class that keeps knowledge of exactly what’s going on away from things that shouldn’t know about it. Didn’t get much else done this hour – was too busy eating lasagne. Going to take an break now and start again in an hour.

• Fixed up AmazonSearcher so it passes its unit tests.
• Set search to redirect to the search form if no query is supplied.
• Created Searcher which holds all searchers and amalgamates results.
• Had lunch.

Hour Seven

Added a stylesheet so I can start to make it look like it should. Added in the concept of a User who may or may not be registered. Plus Location (get from IP initially, can be changed by the user). Everything is a bit haywire at the moment as I assimilate these changes.

Hour Eight

Had to take some time out to do some other things but now have a styled-up, eBay and Amazon searching app with a concept of User and Location. Upon initial visit you are silently logged in as a guest with a location guessed from your IP address. Eventually only sources that are relevant to your location will be searched.

I need to build the functionality to change your location and to register with the site. I also need to build in the concept of location to the existing eBay and Amazon searching libraries.

Still loads to do but lumbering towards a kind of “prototype” state.

Hours Nine and Ten

Last night I got the application running off the root of the site (localhost/ instead of localhost/finder/) and solidified a few architectural decisions.

This morning I’ve followed through on the architectural decisions and written any missing unit tests. There’s still one not passing to do with looking up location from IP address.

• Now works off site root, not /finder/blah
• Architecture of data sources, locations, etc. now clearer.

Hours Eleven and Twelve

Fixed up a lot of little bits and made prices/currency work how it should. Want to get on to persisting searches soon but lots of little things to do first.

• Fix failing unit test
• Sort results by price, lowest first
• Added concept of currency, extended Money from money module

Hours Thirteen and Fourteen

Made quite a lot of progress yesterday afternoon but then had to go out for a friend’s birthday. I’m not going to make 30 hours over the long weekend but it’ll be good to see how far I can get today. Not sure if I will try and make something live tonight or finish off the 30 hours another day – I’ll decide that later on.

• Make change location page work properly.
• Get Amazon and eBay searchers to respect user location and search the appropriate site.
• User registration.

Hour Fifteen – Half Way

Made good progress for an hour but had to stop working on it to produce a website at very short notice for the Lady Margaret short film.

I’ve used up half my allotted hours and the app is fairly functional. The big thing now is to add the persistent part of persistent search – to let users save their searches and be notified when new items become available. I also want to expand it to more data sources like abebooks.com. I’m not sure when I’ll get to do the other 15 hours but I definitely want to get something live quite soon.

• Added France, Germany, Japan, Ireland.
• Fixed for different currencies in Amazon searcher.
• Proper message if there are no results.
• Handle SearchErrors from Amazon.

I’ve revamped londoncinema.bluebones.net.

As well as a new design it has much-improved cinema and film pages, movie posters, trailers and imdb.com ratings.

Search is now even better because as well as taking your location it also uses imdb.com ratings to push better films higher up. With so many films showing it’s so easy to miss something good and with so many cinemas in London it’s so much better than trawling through cinema-by-cinema.

A few weeks ago I decided to teach myself Graphic Design. I’m keeping a little blog with my notes on the subject at design.bluebones.net.

Don’t seem to have one: Javascript, C, C#

Not beautiful but it does effectively flush all cookies:

ClientCookie.install_opener(ClientCookie.build_opener())

I’ve added the most-requested feature to my Fixtures Generator.

You can now enter a newline-delimited list of team names in the initial form and get back team names in the fixture list.

Please let me know if you use it, especially if you have any problems with it.

Fixtures Generator

A formatting function to go from numbers like 333999333.33 to “333,999,999.33” in Haskell. Copes with negative numbers and rounds to 2 dp (easy to add a paramater for that should you wish).

Examples:

*Main> formatDecimal 44
"44.00"
*Main> formatDecimal 94280943.4324
"94,280,943.43"
*Main> formatDecimal (-89438.329)
"-89,438.33"

import Data.Graph.Inductive.Query.Monad (mapFst)
import List
import Text.Printf

formatDecimal d
    | d < 0.0   = "-" ++ (formatPositiveDecimal (-d)) 
    | otherwise = formatPositiveDecimal d 
    where formatPositiveDecimal = uncurry (++) . mapFst addCommas . span (/= '.') . printf "%0.2f" 
          addCommas = reverse . concat . intersperse "," . unfoldr splitIntoBlocksOfThree . reverse 
          splitIntoBlocksOfThree l = case splitAt 3 l of ([], _) -> Nothing; p-> Just p

If you know a simpler way or spot anything that should be done differently, please add a comment.

After investigating a lot of blind alleys I think I’ve solved a problem with pushing and pulling to and from a remote repository being very slow. darcs was issuing hundreds of scp commands just to push up a one line change. This seemed to be related to the _darcs/inventory file being a couple of thousand lines long (should be much smaller) on the remote repo.

In the final analysis my advice is very simple:

• Make sure you have recently done a darcs tag
• Issue a darcs optimize --reorder-patches command (the reorder patches part may be unnecessary and cause it to be much slower, but it worked for me although it took three hours).

This reduced push/pull time from 3 minutes to nearer 3 seconds.

After reading A World of Endless Advertisements I thought it would be interesting to see the AdBlock Plus version next to the unblocked version.

How “ad-supported content” is supported without ads, I don’t know. But as Banksy says:

“Any advertisement in public space that gives you no choice whether you see it or not is yours. It belongs to you. It’s yours to take, re-arrange and re-use. Asking for permission is like asking to keep a rock someone just threw at your head.”

(Faintly amusingly I actually had difficulty writing this post because AdBlock blocked the images I was using because their name contained the word “ads”.)

I had to integrate a haskell program into an existing make-based intrastructure this week. After a few false starts I managed to proxy to cabal through make with the following Makefile. I also added a feature to copy the binary created to a “bin” folder at the same level as the src folder. I’ve used the Haq example name from How to Write a Haskell Program and my cabal setup is similar to the one described there. Trying to compile Setup.lhs or anything more complicated is a mistake and I am assured that runhaskell is portable.

all:
	runhaskell Setup.lhs configure
	runhaskell Setup.lhs build
	cp dist/build/haq/haq ../bin/

clean: 
	runhaskell Setup.lhs clean
	rm -rf ../bin/*