“Perfect” Libraries

Sometimes you use a third party library and the interface is so well designed it’s just effortless. Something that would have been gnarly and murky becomes simple. The kind of library that gets ported to multiple languages because everyone wants access to it.

One slightly obscure example is feedparser, (originally) Mark Pilgrim’s python2 library for reading Atom and RSS feeds. Hiding all this nonsense:

behind a simple interface.

import feedparser 
d = feedparser.parse('http://www.reddit.com/r/python/.rss') 
print(d['feed']['title'])
>>> Python
print d.feed.subtitle 
>>> news about the dynamic, interpreted, interactive, object-oriented, extensible programming language Python 
print d.headers           
>>>  {'content-length': '5393', 'content-encoding': 'gzip', 'vary': 'accept-encoding', 'server': "'; DROP TABLE servertypes; --", 'connection': 'close', 'date': 'Mon, 14 Oct 2013 09:13:34 GMT', 'content-type': 'text/xml; charset=UTF-8'}

Another library that has the same simplicity is Mustache logic-less templates. This one has been ported to literally dozens of languages. Every template I ever worked on was kind of a mess until I found Mustache. It’s actually the restrictions here that make it sing.

Hello {{name}} 
You have just won {{value}} dollars! 
{{#in_ca}} Well, {{taxed_value}} dollars, after taxes. {{/in_ca}}

Some other examples:

  • web.py – Dead simple web framework
  • BeautifulSoup – HTML/XML parser
  • requests – Python library for HTTP
  • humps – Underscore-to-camelCase converter (and vice versa) for strings and object keys in JavaScript (has been ported to Python as pyhumps).
  • Markdown – Text format with HTML representation that has taken over the web due to its simplicity and usefulness compared to actual HTML

Do you know any “perfect” libraries?

GPX to PostGIS, PostGIS to GPX

With ogr2ogr.

export CONN_STRING="host=localhost dbname=DATABASE user=USERNAME password=PASSWORD port=5432"
# Import
ogr2ogr -append -f PostgreSQL PG:dbname=DATABASE_NAME /path/to/your.gpx
# Export
ogr2ogr -f gpx -nlt MULTILINESTRING /path/to/output/tracks.gpx PG:"$CONN_STRING" "tracks(wkb_geometry)"
ogr2ogr -f gpx -nlt MULTILINESTRING /path/to/output/routes.gpx PG:"$CONN_STRING" "routes(wkb_geometry)"
ogr2ogr -f gpx -nlt POINT /path/to/output/waypoints.gpx PG:"$CONN_STRING" "waypoints(wkb_geometry)"

The wkb_geometry references can be replaced with full SQL statements as required.

Short Variable Names in Go

In a recent code review my colleague took issue with the following code.

func Enqueue(properties Properties) (err error) {
	logger := logging.GetLogger(ctx)
	bs, err := json.Marshal(properties)
	if err != nil {
		logger = logger.With().Err(err).Logger()
	} else {
		logger = logger.With().RawJSON("properties", bs).Logger()
	}
        … go on to log some stuff and enqueue the supplied event with the supplied properties …
}

Specifically the question was around whether `bs` was a reasonable name for the variable holding the JSON version of the properties. My counterargument was that short names are better than long names when well understood and/or short in scope. And that Go has a C influence and favors short variable names which you can see in both the standard library and its examples. The Go encoding/json library calls []byte variously src and data (code), b, j and text (examples) – https://golang.org/pkg/encoding/json/

My colleague said it took them longer 0s to understand the var so they called it out as a nit (not a blocker) and that they care more about knowing what is contained within than if it is `[]byte` or not.

I ended up renaming it `propertiesJSON`. It did start a discussion about short variable names in general and in Go in particular. I’m still not sure how I feel about it. I did find some reading that seemed relevant though.

Notes on Programming in C (Variable names)

Variable names in Go should be short rather than long. This is especially true for local variables with limited scope. Prefer c to lineCount. Prefer i to sliceIndex.

Go Code Review Comments (Variable Names)

Local variables. Keep them short; long names obscure what the code does … Prefer b to buffer.

What’s In a Name? (Local Variables)

Share Clipboard Between Terminal (zsh) and OSX

### System-wide Clipboard mostly from https://gist.github.com/welldan97/5127861

pb-kill-line () {
  zle kill-line
  echo -n $CUTBUFFER | pbcopy
}

pb-backward-kill-line () {
  zle backward-kill-line
  echo -n $CUTBUFFER | pbcopy
}

pb-kill-whole-line () {
  zle kill-whole-line
  echo -n $CUTBUFFER | pbcopy
}

pb-backward-kill-word () {
  zle backward-kill-word
  echo -n $CUTBUFFER | pbcopy
}

pb-kill-word () {
  zle kill-word
  echo -n $CUTBUFFER | pbcopy
}

pb-kill-buffer () {
  zle kill-buffer
  echo -n $CUTBUFFER | pbcopy
}

pb-copy-region-as-kill-deactivate-mark () {
  zle copy-region-as-kill
  zle set-mark-command -n -1
  echo -n $CUTBUFFER | pbcopy
}

pb-yank () {
  CUTBUFFER=$(pbpaste)
  zle yank
}

zle -N pb-kill-line
zle -N pb-backward-kill-line
zle -N pb-kill-whole-line
# This is too extreme - I often want to wrangle a commandline then paste into it.
#zle -N pb-backward-kill-word
#zle -N pb-kill-word
zle -N pb-kill-buffer
zle -N pb-copy-region-as-kill-deactivate-mark
zle -N pb-yank

bindkey '^K'   pb-kill-line
bindkey '^U'   pb-backward-kill-line
#bindkey '\e^?' pb-backward-kill-word
#bindkey '\e^H' pb-backward-kill-word
#bindkey '^W'   pb-backward-kill-word
#bindkey '\ed'  pb-kill-word
#bindkey '\eD'  pb-kill-word
bindkey '^X^K' pb-kill-buffer
bindkey '\ew'  pb-copy-region-as-kill-deactivate-mark
bindkey '\eW'  pb-copy-region-as-kill-deactivate-mark
bindkey '^Y'   pb-yank

Calculating Swiss Record Required to Reach Elimination Rounds

Here’s some Python that calculates how many players will reach each record in a Swiss tournament with a Top 8 or similar cut.

from typing import Sequence

# Math from https://www.mtgsalvation.com/forums/magic-fundamentals/magic-general/325775-making-the-cut-in-swiss-tournaments
def swisscalc(num_players: int, num_rounds: int, num_elimination_rounds: int) -> Sequence[int]:
    num_players_in_elimination_rounds = 2 ** num_elimination_rounds
    base = num_players / (2 ** num_rounds)
    num_players_by_losses = [0] * (num_rounds + 1)
    multiplier = 1.0
    total_so_far = 0
    record_required = None
    for losses in range(0, num_rounds + 1):
        wins = num_rounds - losses
        numerator = wins + 1
        denominator = losses
        if denominator > 0:
            multiplier *= (numerator / denominator)
        num_players_by_losses[losses] = base * multiplier
        if not record_required and num_players_in_elimination_rounds:
            total_so_far += num_players_by_losses[losses]
    return num_players_by_losses

Example usage:

$ python3
>>> rounds = 4
>>> r = swisscalc(24, rounds, 3)
>>> for losses in range(len(r)):
...     print(f'{r[losses]} players at {rounds - losses}–{losses}')
... 
1.5 players at 4–0
6.0 players at 3–1
9.0 players at 2–2
6.0 players at 1–3
1.5 players at 0–4

git popclean

If you git stash when you have a bunch of local files that are ignored git stash pop will refuse to un-stash your saved changes. This command cleans that up.


git stash pop 2>&1 | grep already | cut -d' ' -f1 | xargs rm && git stash pop

Make a PUT request in Go (with JSON body)

package main

import (
	"bytes"
	"encoding/json"
	"fmt"
	"net/http"
	"time"
)

type MyRequest struct {
	Name  string `json:"name"`
	Color string `json:"color"`
	Size  int    `json:"size"`
}

type MyResponse struct {
	Status string `json:"status"`
}

func doRequest(httpMethod string, address string, requestBody MyRequest, responseBody *MyResponse) (err error) {
	j, err := json.Marshal(requestBody)
	if err != nil {
		return
	}
	req, err := http.NewRequest(httpMethod, address, bytes.NewReader(j))
	if err != nil {
		return
	}
	req.Header.Set("Content-type", "application/json")
	client := http.Client{Timeout: time.Second * 10}
	resp, err := client.Do(req)
	if err != nil {
		return
	}
	defer resp.Body.Close()
	if resp.StatusCode >= 400 {
		return fmt.Errorf("Request failed with status %d", resp.StatusCode)
	}
	err = json.NewDecoder(resp.Body).Decode(responseBody)
	if err != nil {
		return
	}
	return
}

func main() {
	responseBody := new(MyResponse)
	err := doRequest("PUT", "https://example.com/endpoint", MyRequest{Name: "bakert", Color: "red", Size: 10}, responseBody)
	if err != nil {
		fmt.Println("Failed", err)
	} else {
		fmt.Println("Status", responseBody.Status)
	}
}

Keeping CDN-based js dependencies up to date

For python dependencies you can use a simple requirements.txt file and something like requires.io to keep all your dependencies at latest stable release automatically.

npm and webpack and friends can do a fine job of keeping your js dependencies up to date if you are prepared to bundle them into your js.

But what if you want to use the publicly-hosted CDN versions? There doesn’t seem to be anything available.

This is what I came up with. I’m not in love with it but it works ok so far!

It reads a file that looks like this:

jquery
jquery.hoverIntent
jquery.tablesorter
jquery.tablesorter:jquery.tablesorter.widgets
moment.js
moment-timezone:moment-timezone-with-data
Chart.js
react
react-dom

and emits a file that looks like this:

<script defer src="//cdnjs.cloudflare.com/ajax/libs/jquery/3.4.1/jquery.min.js"></script>
<script defer src="//cdnjs.cloudflare.com/ajax/libs/jquery.hoverintent/1.10.0/jquery.hoverIntent.min.js"></script>
<script defer src="//cdnjs.cloudflare.com/ajax/libs/jquery.tablesorter/2.31.1/js/jquery.tablesorter.min.js"></script>
<script defer src="//cdnjs.cloudflare.com/ajax/libs/jquery.tablesorter/2.31.1/js/jquery.tablesorter.widgets.min.js"></script>
<script defer src="//cdnjs.cloudflare.com/ajax/libs/moment.js/2.24.0/moment.min.js"></script>
<script defer src="//cdnjs.cloudflare.com/ajax/libs/moment-timezone/0.5.26/moment-timezone-with-data.min.js"></script>
<script defer src="//cdnjs.cloudflare.com/ajax/libs/Chart.js/2.8.0/Chart.min.js"></script>
<script defer src="//cdnjs.cloudflare.com/ajax/libs/react/16.10.2/umd/react.production.min.js"></script>
<script defer src="//cdnjs.cloudflare.com/ajax/libs/react-dom/16.10.2/umd/react-dom.production.min.js"></script>

Here’s the code, which lives at https://github.com/PennyDreadfulMTG/Penny-Dreadful-Tools/blob/master/maintenance/client_dependencies.py:

import re
import subprocess
from typing import List

from shared import fetch_tools
from shared.pd_exception import DoesNotExistException

PATH = 'shared_web/templates/jsdependencies.mustache'

def ad_hoc() -> None:
    tags = [fetch_script_tag(library) + '\n' for library in get_dependencies()]
    output = ''.join(tags)
    write_dependencies(output)
    send_pr_if_updated()

def get_dependencies() -> List[str]:
    f = open('shared_web/jsrequirements.txt', 'r')
    return [line.strip() for line in f.readlines()]

def write_dependencies(s: str) -> None:
    f = open(PATH, 'w')
    f.write(s)

def send_pr_if_updated() -> None:
    subprocess.call(['git', 'add', PATH])
    if subprocess.call(['git', 'commit', '-m', 'Update client dependencies.']) == 0:
        subprocess.call(['git', 'push'])
        subprocess.call(['hub', 'pull-request', '-b', 'master', '-m', 'Update client dependencies.', '-f'])

def fetch_script_tag(entry: str) -> str:
    parts = entry.split(':')
    library = parts[0]
    file = parts[0] if len(parts) == 1 else parts[1]
    info = fetch_tools.fetch_json(f'https://api.cdnjs.com/libraries/{library}')
    version = info.get('version')
    if not version and library.lower() != library:
        library = library.lower()
        info = fetch_tools.fetch_json(f'https://api.cdnjs.com/libraries/{library}')
        version = info.get('version')
    if not version:
        raise DoesNotExistException(f'Could not get version for {library}')
    path = None
    for a in info['assets']:
        if a.get('version') == version:
            for f in a['files']:
                if minified_path(f, file):
                    path = f
                    break
                if unminified_path(f, file):
                    path = f
    if not path:
        raise DoesNotExistException(f'Could not find file for {library}')
    return f'<script defer src="//cdnjs.cloudflare.com/ajax/libs/{library}/{version}/{path}"></script>'

def minified_path(path: str, library: str) -> bool:
    return test_path(path, library, '.min')

def unminified_path(path: str, library: str) -> bool:
    return test_path(path, library)

def test_path(path: str, library: str, required: str = '') -> bool:
    # CommonJS libs get us the error 'require is not defined' in the browser. See #6731.
    if 'cjs/' in path:
        return False
    name_without_js = library.replace('.js', '')
    regex = fr'{name_without_js}(.js)?(.production)?{required}.js

    return bool(re.search(regex, path, re.IGNORECASE))

Set Up a New Mac

Updated from a few years ago: http://bluebones.net/2015/12/things-i-absolutely-must-do-on-a-new-mac/

  • Upgrade to latest OSX.
  • Restore home directory from Backblaze. (Or at least dotfiles, .ssh dir, .vim dir, bin directory, ~/Library/Application Support/Alfred 4/*, pwd.dsv.gpg, business, travel)
  • Dock: remove everything, hiding on, magnification on.
  • Sublime Text 3 (and add license key, set “trim_trailing_white_space_on_save”: true in sublime prefs).
  • Solarized theme for Sublime Text 3 (after installing Package Control).
  • subl commandline sublime.$ ln -s “/Applications/Sublime Text.app/Contents/SharedSupport/bin/subl” ~/bin/subl
  • Sublime git ignorer (to have ST3 respect .gitignore in searches, etc.) via Package Control.
  • Hide Desktop icons.
    $ defaults write com.apple.finder CreateDesktop false
    $ killall Finder
  • Alfred (and give it Accessibility access, add license key).
  • Solarized theme for Terminal.
  • “Use Option as Meta Key” in Terminal to make left Alt work as Esc.
  • Use Ctrl-f7 or the Keyboard, Shortcuts in System Preferences to give Full Keyboard Access so that tab takes you to every button in a dialog, etc.
  • Discord.
  • Homebrew. (Installs OSX commandline tools.)
  • Change all .txt to open with Sublime Text.
  • Show hidden files in Finder. Cmd-Shift-.
  • WhatsApp.
  • 1Password.
  • Wine + MTGO: https://github.com/pauleve/docker-mtgo/wiki/macOS:-installing-MTGO-using-Wine
  • Chrome. (Log in to Chrome to get uBlock Origin and other extensions.)
  • Skitch.
  • mysql/mariadb (via homebrew).
  • rtm-cli (via homebrew).
  • Backblaze.