Random rows in MySQL with Reasonable Performance

Almost every answer on Stack Overflow for this is terrible on moderately complex or large real data – https://stackoverflow.com/questions/4329396/mysql-select-10-random-rows-from-600k-rows-fast.


This means executing your query for the entire resultset and then ordering it and then chopping off the number you need. This can have truly dire performance.

Bad: SELECT MAX(id) FROM table
Don’t pick random numbers from 1 to MAX(id) – deleted rows will be null or result in you getting less rows than you want. Who says your id is even sequential/numeric?

OK: SELECT id FROM table

Then pick N at random (removing those already chosen if you don’t want duplicates) from the resultset in your chosen programming language.

If you just want one row from the db you can do this in SQL only as shown here: https://stackoverflow.com/a/31066058/375262. Doing N unique rows this way is left as an exercise for the reader.

If you have a gargantuan resultset even SELECT COUNT(*) might be slow. What would you do then?

Top 50 Rising Programming Technologies

(Based on Stack Overflow tag count and upward trend as proportion of Stack Overflow questions. So it might just be the 50 most difficult-to-learn rising programming technologies!)

The List

  1. Python

    Interpreted high-level programming language for general-purpose programming

  2. React

    JavaScript library for building user interfaces

  3. Laravel

    PHP web framework (“for web artisans”)

  4. Pandas

    Data structures and data analysis tools for Python

  5. TypeScript

    Superset of JavaScript that adds static typing (“JavaScript that scales”)

  6. Amazon Web Services

    On-demand cloud computing platform

  7. API

    The interface between two programs.

  8. Azure

    Cloud computing service

  9. Powershell

    Commandline shell and associated scripting language

  10. Firebase

    Mobile and web application development platform

  11. Selenium
  12. Spring Boot
  13. Docker
  14. React Native
  15. DataFrame
  16. Unity 3D
  17. Elasticsearch
  18. Matplotlib
  19. Go
  20. Jenkins
  21. Selenium Web Driver
  22. Gradle
  23. Machine Learning
  24. Amazon S3
  25. vue.js
  26. ggplot2
  27. Flask
  28. ASP.NET Core
  29. npm
  30. Webpack
  31. Mongoose
  32. tkinter
  33. Google Apps Script
  34. Web Scraping
  35. Spring Security
  36. filter
  37. https
  38. Woo Commerce
  39. Xamarin Forms
  40. Web Socket
  41. Android Recycler View
  42. Kotlin
  43. Redux
  44. Google Sheets
  45. Excel Formula
  46. SASS
  47. Hive
  48. Java 8
  49. Redis
  50. CMake

The Top Ten

Chart of the top 10:

The Top Ten Without Python

Python dwarfs everything else so here’s a look without Python:

The Nearly Men

These tags were eliminated from the list solely on the basis of a 2018-only downward trend: R (would have been 2nd), Node.js (2nd), PostgreSQL (4th), numpy (12th), Express (14th), Apache Spark (14th), Tensorflow (18th), nginx (20th), Github (21st), Amazon EC2 (31st), ECMAScript 6 (39th), ffmpeg (46th)

Programming Languages That Make the List

  1. Python
  2. TypeScript
  3. Go
  4. Kotlin

A Rising Python Lifts All Python Libraries

python-3.x actually makes second place on the list but I rolled it into Python rather than make a redundant entry.

Django and Django Models were eliminated from the list despite being on an upward trend because they have not yet exceeded their previous peak in 2010. This resurgence, Pandas in fourth place and the presence of tkinter on the list speaks to the general rising of Python.


Load all tags on StackOverflow by count descending.

Put each of them into StackOverflow Trends and judge by eye if they are currently trending up.

Where Do These Technologies Sit in the Overall List?

In the whole list of tags by count Python is sixth overall, Firebase (10th place in this list) is 90th, and CMake (50th) is 442nd.

Minimal Implementation of graphql-kotlin

This will bring up a GraphQL endpoint at http://localhost:8080/graphql and the GraphiQL query tool at http://localhost:8080/graphiql if you run ./gradlew bootRun


plugins {
    id("io.spring.dependency-management") version ("1.0.6.RELEASE") // Pull in dependencies automatically.
    id("org.jetbrains.kotlin.jvm") version ("1.3.10")
    id("org.jetbrains.kotlin.plugin.spring") version ("1.3.10")
    id("org.springframework.boot") version ("2.1.0.RELEASE")

tasks.withType(org.jetbrains.kotlin.gradle.tasks.KotlinCompile::class.java).all {
    kotlinOptions {
        freeCompilerArgs = listOf("-Xjsr305=strict") // Enable strict null safety.
        jvmTarget = "1.8"

repositories {

dependencies {
    implementation("com.expedia.www:graphql-kotlin:0.0.23") // Generate GraphQL schema directly from code.
    implementation("com.graphql-java-kickstart:graphiql-spring-boot-starter:5.1") // Get the /graphiql page for free.


package {yourpackagehere}

import com.expedia.graphql.schema.SchemaGeneratorConfig
import com.expedia.graphql.TopLevelObjectDef
import com.expedia.graphql.toSchema
import com.fasterxml.jackson.module.kotlin.KotlinModule
import graphql.schema.GraphQLSchema
import graphql.schema.idl.SchemaPrinter
import graphql.servlet.GraphQLErrorHandler
import graphql.servlet.GraphQLInvocationInputFactory
import graphql.servlet.GraphQLObjectMapper
import graphql.servlet.GraphQLQueryInvoker
import graphql.servlet.ObjectMapperConfigurer
import graphql.servlet.SimpleGraphQLHttpServlet
import javax.servlet.http.HttpServlet
import org.slf4j.LoggerFactory
import org.springframework.boot.autoconfigure.SpringBootApplication
import org.springframework.boot.runApplication
import org.springframework.boot.web.servlet.ServletRegistrationBean
import org.springframework.context.annotation.Bean

class Application {
    private val logger = LoggerFactory.getLogger(Application::class.java)

    fun schema(): GraphQLSchema {
        val schemaConfig = SchemaGeneratorConfig(
            supportedPackages = listOf("{yourpackagehere}"),
            topLevelQueryName = "YourQuery",
            topLevelMutationName = "YourMutation"

        val schema = toSchema(
            queries = listOf(TopLevelObjectDef(YourQuery())),
            mutations = emptyList(),
            config = schemaConfig
        return schema

    fun graphQLObjectMapper(): GraphQLObjectMapper = GraphQLObjectMapper.newBuilder()
            .withObjectMapperConfigurer(ObjectMapperConfigurer { it.registerModule(KotlinModule()) })
            .withGraphQLErrorHandler(GraphQLErrorHandler { it })

    fun graphQLServlet(
        invocationInputFactory: GraphQLInvocationInputFactory,
        queryInvoker: GraphQLQueryInvoker,
        objectMapper: GraphQLObjectMapper
    ): SimpleGraphQLHttpServlet = SimpleGraphQLHttpServlet.newBuilder(invocationInputFactory)

    fun graphQLServletRegistration(graphQLServlet: HttpServlet) = ServletRegistrationBean(graphQLServlet, "/graphql")

fun main(args: Array) {

Chart.js Charts from HTML Data Attributes

Example HTML

    data-labels="[&quot;White&quot;, &quot;Blue&quot;, &quot;Black&quot;, &quot;Red&quot;, &quot;Green&quot;, &quot;Colorless&quot;]" 
    data-series="[13, 11, 5, 2, 2, null]"></canvas>


$('.chart').each(function () {
    var id = $(this).attr("id"),
        type = $(this).data("type"),
        labels = $(this).data("labels"),
        series = $(this).data("series"),
        options = $(this).data("options"),
        ctx = this.getContext("2d");
    new Chart(ctx, {
        'type': type,
        'data': {
            labels: labels,
            datasets: [{
                data: series
        options: options

1205, ‘Lock wait timeout exceeded; try restarting transaction’

Recently we had an issue with very long-lived locks on certain rows in MySQL/MariaDB. Certain people (each row representing a person) would get locked out of updates for hours at a time. Using Flask/uWSGI/MariaDB/mysqldb.

SHOW ENGINE INNODB didn’t report any recent deadlocks or other issues but SHOW PROCESSLIST showed some very long sleeping processes and when these were killed the locks would go away. SELECT trx_started, trx_mysql_thread_id FROM information_schema.innodb_trx t JOIN information_schema.processlist p ON t.trx_mysql_thread_id = p.id; was also interesting showing some unclosed transactiosn that corresponded.

We seem to have cleared the problem up by explicitly closing the database connection on teardown_request.

Atomic Transactional Replacement of a Table in MySQL

Even with AUTOCOMMIT off a DROP TABLE or CREATE TABLE statement will cause an implicit commit in MySQL.

So if you drop your table of (say) aggregated data and then create a new one even if you’re theoretically in a transaction there will be time when clients of the database see no table and time when they see an empty table.

The solution is to use RENAME TABLE.

CREATE TABLE replacement_table (...) AS SELECT ... FROM ...;
CREATE TABLE IF NOT EXISTS current_table (id INT); -- Just in case this is the first run and the table doesn't exist yet so RENAME TABLE doesn't fail.
RENAME TABLE current_table TO old_table, replacement_table TO current_table;

No client of the database will ever see a database that doesn’t contain an existing and populated current_table.

Sharing Terminal kill-ring with system clipboard on OS X

I managed to get vim in the terminal using the system clipboard by using a version compiled with +clipboard and using set clipboard=unnamed in .vimrc. But I wanted to go one further and have the last entry in my emacs-like kill-ring in bash go to the system clipboard too. So that when I hit Ctrl-k on the commandline I can Cmd-v that text into my text editor (or anywhere). Turns out this is pretty tricky.

The best solution I have so far is courtesy of user3439894 on the Apple Stack Exchange. It requires me to use Ctrl-Shift-k to kill instead of Ctrl-k but otherwise does what I want pretty well. You can find the gory details at https://apple.stackexchange.com/a/336361/301884.