Spatial queries with MySQL

2021-03-24T00:00:00+00:00

Photo by Francois Powell, CC BY 2.0, cropped

MySQL is one of the most widely used relational databases. Most PHP websites rely on MySQL for persisting their information, which makes it one of the DB-Engines top four most popular databases along with Oracle, SQL Server, and PostgreSQL.

One of its capabilities that is not very well known is that the engine supports working with spatial data, allowing you to save different shapes (points, lines, polygons) and querying information based on intersections, distances, or overlaps. This capability was included in MySQL a long time ago, but it became easier to use starting in version 5.6, when the distance and point intersection functions were added.

Spatial data can be useful for many needs, including:

Searching for places based on latitude/longitude coordinates
Displaying information and areas as layers on maps
Architecture or home design applications

My first experience with spatial queries was for a weather website I developed that displays local alerts/warnings on a map using MySQL spatial functions to return active weather alerts for a given location, or to inform if lightning has been observed near the user’s current coordinates. So far, MySQL has given me all the resources I need to do such operations with relatively good performance and without needing to write lots of custom code.

Adding spatial information

There are many resources available to import spatial information into our database. From the United States Census Bureau we can find a set of shapefiles with all US states and counties. The Back4App social database platform also has many datasets available to download for free.

Of course, we can also create a table ourselves that contains any kind of spatial information. In the example below, we will create a table named restaurants which will have a name and location (lat/long) geometry for each row.

CREATE TABLE restaurants (
  name VARCHAR(100),
  location GEOMETRY NOT NULL,
  SPATIAL INDEX(location)
);

Note that we are adding an index of type SPATIAL to the location field. This is mainly for performance reasons. Once the table is created, we can populate it with some data using the ST_GeomFromText() function that will convert a string representation of any shape into a geometry:

INSERT INTO restaurants VALUES ('Restaurant 1', ST_GeomFromText('POINT(-26.66115 40.95858)'));
INSERT INTO restaurants VALUES ('Restaurant 2', ST_GeomFromText('POINT(-26.68685 40.93992)'));
INSERT INTO restaurants VALUES ('Restaurant 3', ST_GeomFromText('POINT(-31.11924 42.39557)'));

Querying and filtering with spatial data

Let’s suppose we imported a list of US states from the website above, along with a shape field that holds the geometry associated with each state into a states table. Then, we will be able to get the geometry by just running a SELECT statement:

SELECT state_name, ST_AsText(shape) FROM states ORDER BY state_name;

*-----------------------------*
| state_name | shape          |
*-----------------------------*
| Alabama    | POLYGON((...)) |
| Alaska     | POLYGON((...)) |
| Arizona    | POLYGON((...)) |
| ...                         |
*-----------------------------*

The ST_AsText() function will convert the shape contents into a string representation for us to read and parse, doing the inverse process of the ST_GeomFromText() function we used above. In our application, we can then parse that string and process it the way we need.

Intersecting

For example, we can get a list of states that intersects with a given shape. To do that, we will use the ST_Intersects() function, which will return a boolean value indicating if two shapes intersect or not.

SELECT s.state_name FROM states s WHERE ST_Intersects(s.shape, ST_GeomFromText(?)) ORDER BY s.state_name;

We can define our geometry to be any shape we need. For example, if we want to know which state contains a given point, we can set our parameter to something like 'POINT(40.69 -74.25)' which should return the state of New York.

If we have a polygon, we can set the parameter to specify a polygon shape, i.e. 'POLYGON((40.69 -74.25, 41.10 -74.25, 41.10 -76.11, 40.69 -76.11, 40.69 -74.25))' will mean a rectangle that intersects the states of New York and Pennsylvania. In that case, our query will return two rows with both state names.

Spherical distance

Let’s get back to our restaurants table. If we want to get a list of restaurants that are close to our location, we can use the ST_Distance_Sphere() function that will return a distance in meters between two points in a sphere (defaulting to Earth’s radius):

SELECT * FROM restaurants WHERE ST_Distance_Sphere(location, ST_GeomFromText(?)) <= 10 * 1000 ORDER BY name;

This query will give us a list of restaurants, sorted by name, that are within 10 kilometers from the point we pass as a parameter. We should set the parameter as 'POINT(lat long)' where lat and long represent our current geolocation.

We can use the distance function for many other purposes, like finding people that are close to each other or planning a road trip for the user.

Conclusion

We can avoid writing our own routines to handle spatial data by using MySQL’s built-in spatial types and methods. For most websites, MySQL offers a set of functions that will provide support for most common scenarios. And starting with version 8.0 MySQL includes extended support for geographic and ellipsoid computations.

We should consider that, if we need a complex solution for a robust spatial-based enterprise application, there are alternatives like PostgreSQL’s PostGIS that might be better suited to our purpose. MySQL lacks some advanced features like transformations, custom topology handling, or BRIN indexes, that might have an impact on our development process, depending on our needs.

The image below shows several weather alerts rendered on a Leaflet.js map that were fetched from a field of type GEOMETRY in MySQL:

Resources

There are many other functions we can use to work with our spatial data, from getting the centroid associated to a shape in order to automatically center a map view, to simplifying/optimizing existing geometry. I recommend reviewing the following links to start trying things out:

Containerizing Magento with Docker Compose: Elasticsearch, MySQL and Magento

2020-08-27T00:00:00+00:00

Magento is a complex piece of software, and as such, we need all the help we can get when it comes to developing customizations for it. A fully featured local development environment can do just that, but these can often times be very complex as well. It’d be nice to have some way to completely capture all the setup for such an environment and be able to get it all up and running quickly, repeatably… even with a single command. Well, Docker containers can help with that. And they can be easily provisioned with the Docker Compose tool.

In this post, we’re going to go in depth into how to fully containerize a Magento 2.4 installation for development, complete with its other dependencies Elasticsearch and MySQL. By the end of it, we’ll have a single command that sets up all the infrastructure needed to install and run Magento, and develop for it. Let’s get started.

Magento 2.4 application components

The first thing that we need to know is what the actual components of a Magento application are. Starting with 2.4, Magento requires access to an Elasticsearch service to power catalog searches. Other than that, we have the usual suspects for typical PHP applications. Here’s what we need:

MySQL
Elasticsearch
A web server running the Magento application

In terms of infrastructure, this is pretty straightforward. It would cleanly translate into three separate machines talking to each other via the network, but in the Docker world, each of these machines become containers. Since we need multiple containers for our infrastructure, things like Docker Compose can come in handy to orchestrate the creation of all that. So let’s get to it.

Creating a shared network

Since we want to create three separate containers that can talk to each other, we need to ask the Docker engine to create a network for them. This can be done with this self-explanatory command:

docker network create magento-demo-network

magento-demo-network is the name I’ve chosen for my network but you can choose whatever is most appropriate.

You can run the following command to check your newly created network:

docker network ls

Output usually looks like this:

$ docker network ls
NETWORK ID    NAME                  DRIVER  SCOPE
bd562b9cf5a4  bridge                bridge  local
adb9ec2365c5  host                  host    local
2dba8d97410e  magento-demo-network  bridge  local
c3473c60ed52  none                  null    local

There’s our magento-demo-network network among other networks that Docker creates by default.

Containerizing MySQL

Getting a MySQL instance up and running is super easy these days thanks to Docker. There’s already an official image for MySQL in Docker Hub so we will use that. We can set it up with this command:

docker run -d \
  --name magento-demo-mysql \
  --network magento-demo-network \
  --network-alias mysql \
  -p 3306:3306 \
  -v magento-demo-mysql-data:/var/lib/mysql \
  -e MYSQL_ROOT_PASSWORD=password \
  -e MYSQL_USER=kevin \
  -e MYSQL_PASSWORD=password \
  -e MYSQL_DATABASE=magento_demo \
  mysql:5.7

And just like that, we have a running MySQL instance. Running docker ps can get you a list of currently running containers. The one we just created should show up there.

$ docker ps
CONTAINER ID  IMAGE      COMMAND                 CREATED         STATUS         PORTS                              NAMES
b73739ad5d66  mysql:5.7  "docker-entrypoint.s…"  22 seconds ago  Up 21 seconds  0.0.0.0:3306->3306/tcp, 33060/tcp  magento-demo-mysql

Let’s go through each one of the options from that command now to understand it better.

docker run -d: Runs the container in detached mode. This means that it’s run in the background as a daemon. Control is returned to the console immediately.
--name magento-demo-mysql: This is the name of our container. Normally, Docker will generate random names for containers. In this case, we want to give it a name to refer to it with other Docker commands.
--network magento-demo-network: Tells Docker to run the container as part of the magento-demo-network network that we created earlier. This is the network that we will use for all of our containers.
--network-alias mysql: This is the name of this container within the network. This is how other containers in the network will be able to reference it. We’ll see that come to life a bit later.
-p 3306:3306: Sets up our new MySQL container to allow connections over port 3306. This is MySQL’s default port, which Magento will use to connect to it. This basically says “requests coming over the network to port 3306 of this container are going to be handled by the service installed in this container that listens to port 3306”. That service happens to be MySQL.
-v magento-demo-mysql-data:/var/lib/mysql: Creates a Docker volume. Specifically, we’re setting this one up to store the data files from MySQL. We need to do this so that the data stored in our MySQL container is persisted across shutdowns. magento-demo-mysql-data is the name of the volume and /var/lib/mysql is the directory within the MySQL container where that volume is mounted. In other words, any files stored in that directory are going to be stored within the volume instead. The volume is stored by Docker in the host machine, outside the container. /var/lib/mysql is the default directory where MySQL stores databases.
-e MYSQL_ROOT_PASSWORD=password: Is the password for the root user for MySQL. This is passed into the containerized MySQL via environment variables. Hence the -e option.
-e MYSQL_USER=kevin: Creates a new login in MySQL with kevin as its username.
-e MYSQL_PASSWORD=password: Sets the word password as the password for that kevin user.
-e MYSQL_DATABASE=magento_demo: Creates a database named magento_demo.
mysql:5.7: This is the image that we’re using for our container. 5.7 specifies the version that we want to run. The mysql image in Docker Hub contains a few more versions. Or “tags”, in Docker words.

Connecting to this container

docker ps showed us that our container was running. We can also interact with it. Here are a couple of ways of doing it:

Connecting from within the container

The easiest way of connecting to the MySQL instance is by running mysql CLI client from within the container itself. You can do that with:

docker exec -it magento-demo-mysql mysql -u kevin -p

Here’s how that command works:

docker exec -it is used to run commands inside a container in interactive mode. Just what we need here in this case because we’re running mysql, which is an interactive CLI.
magento-demo-mysql is the name we gave our container in the docker run command from before via the --name magento-demo-mysql option. This is why it’s useful to give names to containers: so we can use them in commands like this.
mysql -u kevin -p is the command that’s run within the container. This is just the usual way of connecting to a MySQL server instance using the mysql CLI client. We use kevin because that’s what we set MYSQL_USER to when we created our container before.

After running the previous command, the console will ask you for your password. We set that to password via MYSQL_PASSWORD so that’s what we need to type in. This will eventually result in the mysql prompt showing up. Run show databases to confirm that the magento_demo database that we specified via MYSQL_DATABASE got created.

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| magento_demo       |
+--------------------+
2 rows in set (0.00 sec)

You can Ctrl + D your way out of that when you’re done exploring the containerized MySQL instance.

Connecting directly from the host machine

We can also connect to the MySQL instance running in the container, directly from our host machine. We can use:

mysql -h localhost -P 3306 --protocol=tcp -u kevin -p

Note that it is required that the mysql CLI client is installed in the host machine for this to work.

Same as before, mysql will ask you for the password and, once typed in, it will give you its prompt.

Containerizing Elasticsearch

Like MySQL, there’s an official Elasticsearch Docker image up in Docker Hub. As a result, getting a working Elasticsearch installation is a piece of cake. It’s done with a command like this:

docker run -d \
  --name magento-demo-elasticsearch \
  --network magento-demo-network \
  --network-alias elasticsearch \
  -p 9200:9200 \
  -p 9300:9300 \
  -e "discovery.type=single-node" \
  elasticsearch:7.8.1

You can validate that the Elasticsearch is running with curl localhost:9200/_cat/health. That should return something like this:

$ curl localhost:9200/_cat/health
1597622135 23:55:35 docker-cluster green 1 1 0 0 0 0 0 0 - 100.0%

Alright! That was easy enough. Again, thanks to Docker, we have an application that’s somewhat complex to set up, up and running in a matter of seconds.

Like before, let’s dissect that command that we used. Very similar to the MySQL one, only with some Elasticsearch specific settings:

docker run -d: Same as with the MySQL container, runs it in detached mode.
--name magento-demo-elasticsearch: Gives the container a friendly name.
--network magento-demo-network: Puts the container in the same network as the rest of our infrastructure.
--network-alias elasticsearch: Is the name by which other containers in the network can refer to this contianer.
-p 9200:9200: Opens port 9200 so that other containers within the network can talk to this one.
-p 9300:9300: Same thing but for a different port.
-e "discovery.type=single-node": Sets up the discovery.type environment variable that the image uses to configure Elasticsearch with.
elasticsearch:7.8.1: Specifies that our container will be running version 7.8.1 of Elasticsearch.

Containerizing Magento

Now this is the step where things get a little bit more involved. Nothing crazy however, so let’s get into it.

The Dockerfile

There’s no image of Magento 2 that would be able to get us up and running as quickly as with MySQL or Elasticsearch, at least not that I could find, so we’re going to have to create our own. We can create our own images with the help of Dockerfiles. A Dockerfile is a file that contains all the specifications needed for a container. The Docker engine uses it to create images which can then be used as basis for running containers.

Here’s a Dockerfile for Magento 2.4 that I came up with:

# /path/to/project/Dockerfile

# Our image is based on Ubuntu.
FROM ubuntu

# Here we define a few arguments to the Dockerfile. Specifically, the
# user, user id and group id for a new account that we will use to work
# as within our container.
ARG USER=docker
ARG UID=1000
ARG GID=1000

# Install PHP, composer and all extensions needed for Magento.
RUN apt-get update && apt-get install -y software-properties-common curl

RUN add-apt-repository ppa:ondrej/php
RUN apt-get update && apt-get install -y php
RUN apt-get update && apt-get install -y \
    php-mysql php-xml php-intl php-curl \
    php-bcmath php-gd php-mbstring php-soap php-zip \
    composer

# Install Xdebug for a better developer experience.
RUN apt-get update && apt-get install -y php-xdebug
RUN echo "xdebug.remote_enable=on" >> /etc/php/7.4/mods-available/xdebug.ini
RUN echo "xdebug.remote_autostart=on" >> /etc/php/7.4/mods-available/xdebug.ini

# Install the mysql CLI client.
RUN apt-get update && apt-get install -y mysql-client

# Set up a non-root user with sudo access.
RUN groupadd --gid $GID $USER \
    && useradd -s /bin/bash --uid $UID --gid $GID -m $USER \
    && apt-get install -y sudo \
    && echo "$USER ALL=(root) NOPASSWD:ALL" > /etc/sudoers.d/$USER \
    && chmod 0440 /etc/sudoers.d/$USER

# Use the non-root user to log in as into the container.
USER ${UID}:${GID}

# Set this as the default directory when we connect to the container.
WORKDIR /workspaces/magento-demo

# This is a quick hack to make sure the container has something to run
# when it starts, preventing it from closing itself automatically when
# created. You could also remove this and run the container with `docker
# run -t -d` to get the same effect. More on `docker run` further below.
CMD ["sleep", "infinity"]

Feel free to go through the comments in the file above for more details, but essentially, this Dockerfile describes what a machine ready to run Magento should look like. It’s got PHP and all the necessary extensions, Xdebug, and Composer. It also includes the mysql CLI client.

Importantly, it allows for creating a user account with sudo access. Later, we’ll use this capability to create a user account, inside the container that mimics the one we’re using in our host machine, effectively using the same user both inside and outside the container. The purpose of this is to make it possible to work on the Magento source code files from inside the container without having to deal with Linux permission issues when we try to do the same from outside the container (that is, directly via the host machine).

The image

Alright, now that we have our image defined in the form of our Dockerfile, let’s create it. To do that, we go into our project directory, create a new file named Dockerfile:

cd /path/to/project
touch Dockerfile

Then use a text editor to save the contents from above into it, and finally run this command:

docker build \
  --build-arg USER=kevin \
  --build-arg UID=$(id -u) \
  --build-arg GID=$(id -g) \
  -t magento-demo-web .

Here’s what this all means:

docker build: Is the command to build images from Dockerfiles.
--build-arg USER=kevin: Specifies the username for the account with sudo access that we will log into our container as. I’ve chosen kevin here but you should use the one you’re logged in as on your machine.
--build-arg UID=$(id -u): Uses the id -u to pass in the Id of the currently logged in user.
--build-arg GID=$(id -g): Uses the id -g to pass in the Group Id of the currently logged in user.
-t magento-demo-web .: Specifies the name of the resulting image to be magento-demo-web. The . is a reference to the current working directory from where we’re running the command, which is where our Dockerfile is located.

Run docker image ls and you should see our new home grown magento-demo-web image along with the other ones that we’ve downloaded from Docker Hub:

REPOSITORY        TAG     IMAGE ID      CREATED         SIZE
magento-demo-web  latest  90d311df434f  22 minutes ago  452MB
mysql             5.7     718a6da099d8  12 days ago     448MB
ubuntu            latest  1e4467b07108  3 weeks ago     73.9MB
elasticsearch     7.8.1   a529963ec236  3 weeks ago     811MB

The container

Ok, now that we have an image that’s capable of running Magento, let’s put it to work by creating a container based on it. We do that with:

docker run -d \
  --name magento-demo-web \
  --network magento-demo-network \
  --network-alias web \
  -p 5000:5000 \
  -v ${PWD}:/workspaces/magento-demo \
  magento-demo-web

Line by line, this is telling the Docker engine to:

docker run -d: Run the container in detached mode. You could also add the -t argument which makes sure the container stays up and running even if there’s no program or service running within it. We don’t need that in this case though, because we defined our Dockerfile with that nifty sleep infinity command.
--name magento-demo-web: Set the name of our container to magento-demo-web.
--network magento-demo-network: Make our container part of the same network as the MySQL and Elasticsearch ones.
--network-alias web: Set our container’s name within the network.
-p 5000:5000: Open port 5000 to access our soon-to-be-running Magento app.
-v ${PWD}:/workspaces/magento-demo: Create a new volume that makes our current working directory the same as the /workspaces/magento-demo directory within the container. This is where we’ll store all the Magento files. Binding these directories makes it possible to access and modify the Magento files both from the container and from the host machine. This just makes things easier and more convenient for development purposes.
magento-demo-web: Use this image.

Running docker container ls will show a list of all running containers, including the one we just created:

docker container ls
CONTAINER ID  IMAGE                COMMAND                 CREATED        STATUS        PORTS                                           NAMES
4af35c42e0bb  magento-demo-web     "/bin/bash"             5 minutes ago  Up 5 minutes  0.0.0.0:5000->5000/tcp                          magento-demo-web
6c5ea65a7bd6  elasticsearch:7.8.1  "/tini -- /usr/local…"  2 hours ago    Up 2 hours    0.0.0.0:9200->9200/tcp, 0.0.0.0:9300->9300/tcp  magento-demo-elasticsearch
b73739ad5d66  mysql:5.7            "docker-entrypoint.s…"  3 hours ago    Up 3 hours    0.0.0.0:3306->3306/tcp, 33060/tcp               magento-demo-mysql

Connecting to the container

With the container up and running, we can connect to it with:

docker exec -it magento-demo-web bash

You may remember this as the same command we used before to connect to the MySQL container. This time, however, we’re using it to connect to our magento-demo-web container, referenced by the name we gave it, and running bash on it in order to open a shell.

After that, a prompt like this should show up:

kevin@4af35c42e0bb:/workspaces/magento-demo$

We’re now inside our container. Notice how we’re automatically taken to /workspaces/magento-demo. This is just like we specified in our Dockerfile with the WORKDIR command. Feel free to run php -v or composer -V to validate that the setup from our Dockerfile got all the way into our container:

kevin@4af35c42e0bb:/workspaces/magento-demo$ php -v
PHP 7.4.9 (cli) (built: Aug  7 2020 14:30:01) ( NTS )
Copyright (c) The PHP Group
Zend Engine v3.4.0, Copyright (c) Zend Technologies
    with Zend OPcache v7.4.9, Copyright (c), by Zend Technologies
    with Xdebug v2.9.6, Copyright (c) 2002-2020, by Derick Rethans
kevin@4af35c42e0bb:/workspaces/magento-demo$ composer -V
Composer 1.10.1 2020-03-13 20:34:27

Talking to other containers in the network

We also need to validate that our containers are actually able to talk to each other via the network that we set up. If all went according to plan, still from within our magento-demo-web container, this command should open a mysql session:

mysql -h mysql -u kevin -p

Notice how this time we don’t use localhost or 127.0.0.1 to connect to our MySQL instance. This time, we use mysql. This is the network alias we gave out MySQL container, so this is how our magento-demo-web sees it. To magento-demo-web, the MySQL container is just another machine in the same network.

Same deal for the Elasticsearch container. We can do something like this to talk to it:

curl elasticsearch:9200/_cat/health

Again, from the perspective of magento-demo-web, this is just another machine in the network which it can reach by using the elasticsearch network alias that we gave it when creating it.

Installing Magento in our container

Now that we have our environment ready for Magento, let’s install it. First order of business is to create the Composer project:

composer create-project --repository-url=https://repo.magento.com/ magento/project-community-edition ./install

If you’re familiar with Composer, then this should look very familiar to you. This command will download all the Magento files as specified by the magento/project-community-edition project from the https://repo.magento.com/ repository. There are a few gotchas though:

First, Magento is not openly available to download just like that. As such, Composer will ask for authentication in order to do so. Follow this guide to obtain the authentication keys from the Magento Marketplace. When Composer asks for a username, type in the public key; when it asks for password, type in the private key.
Second, you’ll notice that I specified ./install at the end of that command. This is where all the files will be downloaded. I’ve chosen this (an install directory inside our current one) because composer create-project will refuse to download the files in a directory that’s not empty. Ours isn’t, because we’ve got our Dockerfile in it. But that’s nothing to worry about, once Composer finishes downloading everything, we’ll just copy the files over to their rightful location at /workspaces/magento-demo. You can do so with some Linux sorcery like this:

(shopt -s dotglob; mv -v ./install/* .)

This Composer operation will take a good while, but when it’s done, make sure to move all the contents of ./install into /workspaces/magento-demo. We now need to actually install Magento:

bin/magento setup:install \
  --base-url=http://localhost:5000 \
  --db-host=mysql \
  --db-name=magento_demo \
  --db-user=kevin \
  --db-password=password \
  --admin-firstname=admin \
  --admin-lastname=admin \
  --admin-email=admin@admin.com \
  --admin-user=admin \
  --admin-password=admin123 \
  --language=en_US \
  --currency=USD \
  --timezone=America/New_York \
  --use-rewrites=1 \
  --elasticsearch-host=elasticsearch \
  --elasticsearch-port=9200

Even if you have never installed Magento before, the command above should be pretty straightforward. An interesting thing to note is how we’ve set up our database and Elasticsearch settings here:

  --db-host=mysql \
  --db-name=magento_demo \
  --db-user=kevin \
  --db-password=password \

and

  --elasticsearch-host=elasticsearch \
  --elasticsearch-port=9200

--db-host is the hostname of the machine where the MySQL server is running. We use our container’s network alias here. --db-name is the name of the database we created when initializing our container via the MYSQL_DATABASE environment variable. --db-user and --db-password are the credentials for the login that we created in the same manner. --elasticsearch-host is the network alias of our Elasticsearch container, and finally --elasticsearch-port is the port that we configured it to listen to.

As you can see, these are the same settings that we used to configure our MySQL and Elasticsearch containers. So make sure to do the same if you’ve been following along and decided to go with different values.

Once that command is done, we’re ready. We have a working Magento. Try it out by running this:

php -S 0.0.0.0:5000 -t ./pub/ ./phpserver/router.php

And navigating to localhost:5000 in your browser. You should see your empty Magento homepage.

Optional: Installing the sample data

If you’re planning some custom extension, or to just play with Magento to get to know it better, you may want to add some sample data. Luckily, the Magento devs have graciously provided such a thing in the form of a Composer package. If you want, you can install it with this recipe:

bin/magento sampledata:deploy
bin/magento setup:upgrade
bin/magento indexer:reindex
bin/magento cache:flush

bin/magento sampledata:deploy will also ask you for your Magento Makerplace keys so have them ready.

So turn off the built-in PHP server, run these, wait a good while, and fire up the built in server once more. Your Magento app should now have a catalog and all sorts of other data loaded in.

Composing it all together

Now that was a lot. It was much easier than having to set everything up from scratch without Docker, but still, I promised a minimal setup overhead. A single command. With Docker Compose we can do just that.

For containers, the usual workflow is a three step process:

Create the Dockerfile (sometimes omitted if we have a readily available image like it was the case with MySQL and Elasticsearch).
Create or download an image.
Run the container.

Docker Compose can help us by capturing all the settings needed to create containers in a single YAML file; which then can be taken by a CLI tool (i.e. docker-compose) and it can set up the complete infrastructure. This single file is named docker-compose.yml and this is what it may look like for our current setup:

version: "3.8"

# Listing our three containers. Or "services", as known by Docker Compose.
services:
    # Defining our MySQL container.
    # "mysql" will be the network alias for this container.
    mysql:
        image: mysql:5.7
        container_name: magento-demo-mysql
        networks:
            - magento-demo-network
        ports:
            - "3306:3306"
        volumes:
            - magento-demo-mysql-data:/var/lib/mysql
        environment:
            MYSQL_ROOT_PASSWORD: password
            MYSQL_USER: kevin
            MYSQL_PASSWORD: password
            MYSQL_DATABASE: magento_demo

    # Defining our Elasticsearch container
    # "elasticsearch" will be the network alias for this container.
    elasticsearch:
        image: elasticsearch:7.8.1
        container_name: magento-demo-elasticsearch
        networks:
            - magento-demo-network
        ports:
            - "9200:9200"
            - "9300:9300"
        environment:
            discovery.type: single-node

    # Defining our custom Magento 2 container.
    # "web" will be the network alias for this container.
    web:
        # The build section tells Docker Compose how to build the image.
        # This essentially runs a "docker build" command.
        build:
            context: .
            dockerfile: Dockerfile
            args:
                USER: kevin
                UID: 1000
                GID: 1000
        container_name: magento-demo-web
        networks:
            - magento-demo-network
        ports:
            - "5000:5000"
        volumes:
            - .:/workspaces/magento-demo

# The volume that is used by the MySQL container
volumes:
    magento-demo-mysql-data:

# The network where all the containers will live
networks:
    magento-demo-network:

As you can see, most of docker-compose.yml is more or less rewriting the docker run commands in a YAML format. With the exception of the web container/service which includes a build section that reflects the docker build command that was used to take the Dockerfile and turn it into an image.

If you want to try it out, make sure to remove all the infrastructure we’ve created, to avoid any conflicts. You can do so from your host machine with these commands:

docker container rm -f magento-demo-web magento-demo-elasticsearch magento-demo-mysql
docker image rm magento-demo-web
docker network rm magento-demo-network
docker volume rm magento-demo-mysql-data

Make sure you’re in the directory where the Dockerfile lives in the host machine. Then create a new docker-compose.yml file and put all the content above into it. Finally, run:

docker-compose up -d

This will take a little while, but by the end of it, you’ll have a complete infrastructure with the three containers that we’ve created step by step throughout this article. With the docker-compose.yml file, docker-compose up essentially takes care of running all of our docker build and docker run commands.

The -d option means that the the command will run in the background and give you back control of your console. You can also run it without it if you want the console to show the logs from the containers.

You can still see the logs even in detached mode with:

docker-compose logs

You can also inspect the running containers. For that, you can use:

docker-compose ps

Output will look something like this:

$ docker-compose ps
           Name                         Command               State                       Ports                     
--------------------------------------------------------------------------------------------------------------------
magento-demo-elasticsearch   /tini -- /usr/local/bin/do ...   Up      0.0.0.0:9200->9200/tcp, 0.0.0.0:9300->9300/tcp
magento-demo-mysql           docker-entrypoint.sh mysqld      Up      0.0.0.0:3306->3306/tcp, 33060/tcp             
magento-demo-web             sleep infinity                   Up      0.0.0.0:5000->5000/tcp

Notice how docker-compose ps gives us our container names just as we specified them in the docker-compose.yml file.

docker-compose has many other utilities, check them out with docker-compose --help.

Now, same as before, we still need to open a terminal into our Magento container to run some installation commands on it. To do so, we can run the following command:

docker-compose exec web bash

Notice how with docker-compose we refer to the container via its service name. That is, the name we gave the container under the services section of docker-compose.yml.

Of course, we can still use the same command that we used before, when we created our container directly with docker:

docker exec -it magento-demo-web bash

Now, once inside our container we need to install Magento again. Remember that we wiped out all the infrastructure we created manually, so these are fresh new containers; akin to new machines.

If you were running this from scratch you would just go ahead and do…

composer create-project --repository-url=https://repo.magento.com/ magento/project-community-edition ./install

and

(shopt -s dotglob; mv -v ./install/* .)

In this case, however, we already have all the Magento files in our directory, So we can save time and skip this step. We can reuse these files and just run bin/magento setup:install.

But since this is a new Magento installation, we do need to remove the config file before setup:install’ing. So go ahead and…

rm app/etc/env.php

…then:

bin/magento setup:install \
  --base-url=http://localhost:5000 \
  --db-host=mysql \
  --db-name=magento_demo \
  --db-user=kevin \
  --db-password=password \
  --admin-firstname=admin \
  --admin-lastname=admin \
  --admin-email=admin@admin.com \
  --admin-user=admin \
  --admin-password=admin123 \
  --language=en_US \
  --currency=USD \
  --timezone=America/New_York \
  --use-rewrites=1 \
  --elasticsearch-host=elasticsearch \
  --elasticsearch-port=9200

After a while, Magento will be fully installed in our new infrastructure created by Docker Compose and ready to be fired up via the PHP built in server:

php -S 0.0.0.0:5000 -t ./pub/ ./phpserver/router.php

Bonus: Interactive debugging with Visual Studio Code

So this is a fully functioning Magento installation with files that we can edit to our heart’s content. In terms of a “fully featured” development environment, however, we need to spruce it up a bit.

So install VS Code from https://code.visualstudio.com/ and install the Remote Development plugin.

Open a new VS Code window and open the command palette with Ctrl + Shift + P. In there, type in Remote-Containers: Attach to Running Container... and press Enter. In the menu that shows up, select our magento-demo-web container.

That will result in a new VS Code instance that is connected to the container. Open an integrated terminal in VS Code and you’ll see:

Now, install the PHP Debug extension so that we can take advantage of that Xdebug that we installed in our container via our Dockerfile.

Create a new launch configuration for interactive debugging with PHP by clicking on the “Run” button in the action bar to the left (Ctrl + Shift + D also works). Click the “create a launch.json file” link in the pane that appears. Then, in the resulting menu at the top of the window, select the “PHP” option. Here’s a screen capture for guidance:

That will result in a new .vscode/launch.json file created that contains the launch configuration for the PHP debugger.

Now let’s put a breakpoint anywhere, like in line 13 of the pub/index.php file; press the “Start debugging” button in the “Run” pane, near the top left of the screen (making sure that the “Listen to XDebug” option is selected), and start up the PHP built in server from VS Code’s integrated terminal with php -S 0.0.0.0:5000 -t ./pub/ ./phpserver/router.php. Now navigate to localhost:5000 in your browser and enjoy VS Code’s interactive debugging experience:

Summary

Whew! That was quite a bit. In this blog post, we’ve done a deep dive into how to set up all the pieces of a Magento application using Docker containers: MySQL, Elasticsearch, and Magento itself. Then, we captured all that knowledge into a single docker-compose.yml file which can be run with a single docker-compose up command to provision all the infrastructure in our local machine. As a cherry on top, we set up interactive debugging of our brand new Magento application with VS Code. Thanks to the safety net provided by these tools, I feel like I’m ready to really dig into Magento and start developing customizations, or debugging existing websites. If you’ve been following along this far, dear reader, I hope you do too.

How to set up a local development environment for WordPress from scratch

2019-08-07T00:00:00+00:00

I recently got pulled into a project for a client who wanted to have a new WordPress website developed for them. I started by setting up a development environment with the niceties that I’m used to from my other application development work. That is, a development server, interactive debugging, linting, and a good editor.

Another thing that I wanted was not to have to deal with LAMP or WAMP or XAMPP or any of that. I wanted a clean, from scratch installation where I knew and controlled everything that was there. I’ve got nothing against those packages, but I think that, by setting up everything manually, I’d be able to better learn the technology as I would know exactly how everything is set up under the hood. The shortcuts could come later.

Luckily for me, there aren’t many pieces when it comes to setting up a basic, running development environment for WordPress. You only need three things: 1. MySQL, 2. PHP, and 3. WordPress itself. I also wanted a few other goodies and we’ll get there.

Let’s go through the steps that I took to set all of this up:

1. Set up PHP

In Ubuntu, installing PHP is easy enough. Just run the following command:

sudo apt-get install php

After that’s done, run php -v to validate that it was successfully installed. It should result in something like this:

PHP 7.2.19-0ubuntu0.18.04.1 (cli) (built: Jun  4 2019 14:48:12) ( NTS )
Copyright (c) 1997-2018 The PHP Group
Zend Engine v3.2.0, Copyright (c) 1998-2018 Zend Technologies
    with Zend OPcache v7.2.19-0ubuntu0.18.04.1, Copyright (c) 1999-2018, by Zend Technologies

There’s one particular PHP extension that we’re going to need. Let’s install it with:

sudo apt-get install php-mysql

The php-mysql extension is necessary for our PHP installation to interact with MySQL. That’s all that’s needed to run WordPress as far as PHP is concerned.

2. Set up MySQL

WordPress uses MySQL for all of its data storage concerns. So, let’s install it. Again, in Ubuntu, installing and setting up MySQL is super easy. First, we need to run this command:

sudo apt-get install mysql-server

This will install both the MySQL database engine and a command-line client for us to connect to it and do some initial configuration. We now need to log into our freshly installed MySQL instance. But first, make sure that it’s running with:

sudo service mysql start

Now that our instance is running, log into it as root with:

sudo mysql -u root

This will open up the MySQL command-line client where we can do some initial configuration to support WordPress.

We now need to create a new MySQL user that will be used by WordPress to log into the database. You can do so with a command like this from within the MySQL CLI client:

CREATE USER 'wordpress_user'@'localhost' IDENTIFIED BY 'password';

Obviously, choose a username and password that work for you. I like to keep things simple and obvious so that’s what I use. Also obviously, use a strong, unique password in a production environment.

Now, create a new database that will be used by WordPress with:

CREATE DATABASE wordpress_dev;

Again, feel free to choose a name that suits your needs.

Now, we need to allow the user that we created a few steps ago to access and control that new database. Since this is only a dev environment, let’s just give our WordPress user access to everything. This can be done with:

GRANT ALL ON *.* TO 'wordpress_user'@'localhost'

With that, we’re done with MySQL, we can close the CLI client with the exit command.

3. Set up WordPress

Now that we have our prerequisites ready, we can proceed to setting up the actual WordPress site. It turns out, this is pretty easy as well. Get a new directory ready and let’s get started.

First, we need to download the package of WordPress files from the official site. In Ubuntu, this can be done with this command:

wget https://wordpress.org/latest.tar.gz

This will result in a new latest.tar.gz file being created in your directory. Now, extract it with:

tar xvzf latest.tar.gz

Then, enter the new wordpress directory that gets created as a result of the last operation. Explore the wordpress directory and you should be able to see a bunch of wp-* files and directories. These are all the files that WordPress needs to run.

Before running WordPress though, we need to configure it so that it uses the MySQL database that we just created. We do this by specifying that configuration in a wp-config.php file. This file does not exist yet, but we have a wp-config-sample.php file that we can use as a template. Create the new wp-config.php file based on wp-config-sample.php with the following command:

cp wp-config-sample.php wp-config.php

Now, in the new wp-config.php file, put in your MySQL database information. Starting around line 23, it should look like this:

/** The name of the database for WordPress */
define( 'DB_NAME', 'wordpress_dev' );

/** MySQL database username */
define( 'DB_USER', 'wordpress_user' );

/** MySQL database password */
define( 'DB_PASSWORD', 'password' );

/** MySQL hostname */
define( 'DB_HOST', 'localhost' );

/** Database Charset to use in creating database tables. */
define( 'DB_CHARSET', 'utf8' );

/** The Database Collate type. Don't change this if in doubt. */
define( 'DB_COLLATE', '' );

4. Run WordPress

Now we’re finally ready to actually run WordPress. WordPress, as a web application, needs a web server like Apache to run. We don’t have Apache though; what we have is PHP’s built-in development web server. From within our wordpress directory, we can fire up the built-in web server with:

php -S localhost:3000

Now it’s just a matter of navigating to the localhost:3000/wp-admin/install.php page in your browser of choice. This page should show up on screen:

Just follow the steps within the wizard at install.php and your new development WordPress site will be ready to go in no time.

Bonus 1: Set up a linter: PHP Code Sniffer

Prerequisites: Composer and the php-xml extension

You can install Composer by following these instructions. In Ubuntu, installing the php-xml extension can be done with:

sudo apt-get install php-xml

Now, to set up the PHP Code Sniffer linter, just follow these steps, from your wordpress directory:

Install PHP Code Sniffer with composer require --dev squizlabs/php_codesniffer.
Download the WordPress PHP Code Sniffer standards with composer require --dev wp-coding-standards/wpcs.
Install it into phpcs with: vendor/bin/phpcs --config-set installed_paths vendor/wp-coding-standards/wpcs.
Sniff something with vendor/bin/phpcs --standard=WordPress index.php.

If all goes well, you should be able to see reports like this (which is exaggerated for demonstration purposes; your default index.php file will not show as many warnings):

FILE: /home/kevin/projects/random/wordpress/index.php
-----------------------------------------------------------------------------------------
FOUND 5 ERRORS AND 1 WARNING AFFECTING 2 LINES
-----------------------------------------------------------------------------------------
 14 | ERROR   | [x] Expected 1 spaces after opening bracket; 0 found
 14 | ERROR   | [x] Expected 1 spaces before closing bracket; 0 found
 17 | WARNING | [x] "require" is a statement not a function; no parentheses are required
 17 | ERROR   | [x] Expected 1 spaces after opening bracket; 0 found
 17 | ERROR   | [x] Expected 1 spaces before closing bracket; 0 found
 17 | ERROR   | [x] Concat operator must be surrounded by a single space
-----------------------------------------------------------------------------------------
PHPCBF CAN FIX THE 6 MARKED SNIFF VIOLATIONS AUTOMATICALLY
-----------------------------------------------------------------------------------------

Time: 103ms; Memory: 10MB

Bonus 2: Setup interactive debugging with VS Code

Prerequisites: VS Code, XDebug and the PHP Debug VS Code extension

For writing PHP, my editor of choice is VS Code. You can get the editor from the official download site. VS Code has a huge extensions ecosystem where you can find almost anything. Naturally, there’s a debugger for PHP. It’s aptly called PHP Debug. See the instructions on how to set it up. The PHP Debug extension works on top of XDebug, a debugger for PHP. Luckily for us, PHP Debug’s instructions page includes all the details on how to install and set up both itself and XDebug.

If you followed the instructions, you should now have a new .vscode/launch.json file within your wordpress directory with the following contents:

{
   "version": "0.2.0",
   "configurations": [
       {
           "name": "Listen for XDebug",
           "type": "php",
           "request": "launch",
           "port": 9000
       }
   ]
}

This is what’s called a launch configuration in VS Code speak. This tells VS Code’s debugger all the info it needs to attach to a running PHP process. To see it in action, fire up the built-in web development server with:

php -S localhost:3000

Then, click the Start Debugging button. That’s the green triangle icon near the top of the screen, when you select the Debug sidebar in VS Code. It should have the Listen for XDebug option selected.

Now it’s just a matter of putting a breakpoint anywhere within the source code and request the page from your browser. You can set a breakpoint by clicking right next to the line number indicator, within any file. Here, I’ve put a breakpoint on line 14 in index.php:

When the code execution hits the breakpoint, it should stop right there and allow you to inspect variables and the like, just like any other debugger.

And that’s all for now! Hopefully this little write-up can help you jumpstart your next WordPress development project.

Running Magento 2 in Windows with XAMPP

2019-03-22T00:00:00+00:00

Photo by Nicole De Khors · Burst, Some Rights Reserved

Magento is an open source ecommerce platform, written in PHP and relying on MySQL/MariaDB for persistence. According to BuiltWith, Magento is the third most used platform in ecommerce websites. It began its life in 2008 with its first general release, and a major update (Magento 2) was released in 2015.

And now, more than three years after, Magento 1 is slowly dying: There won’t be any more quality fixes or security updates from June 2020, and there won’t be extended support for fixes or new payment methods. So the obvious choice will be Magento 2 from now on.

But is it fully tested yet? Is it stable enough? If we already have a website running with Magento 1, what should we do? Migrating to Magento 2 is not just hitting an “Update” button: Themes are incompatible, most extensions won’t work, and of course, there’s a big set of changes to get familiar with.

So a good approach might be to get a clean Magento 2 version deployed locally, to look what we need to do to get our website updated and running, test the backend, find where the configuration sections are located, and so on. And many business users, and even some developers like myself, have Microsoft Windows installed on our computers.

Environment setup

The environment I used for this testing installation was Windows 10 Professional. As a first step, we’ll need to make sure that localhost is published in our local hosts file:

Navigate to the folder %SystemRoot%\system32\drivers\etc
Backup the existing hosts file
Open a text editor with administrator rights
Open the hosts file
Make sure the first line after the commented (#) lines is 127.0.0.1 localhost and the second is ::1 localhost
Open a cmd window with administrator rights and run the command ipconfig /flushdns

Now we’re ready to install the environment needed to run Magento. I recommend using XAMPP, a free Apache distribution for Windows that includes MariaDB, PHP, and Perl in a single package. Magento 2 currently runs with PHP 7.2 and it will not work with newer versions.

Install XAMPP 7.2 with default settings: apachefriends.org/xampp-files/7.2.15/xampp-win32-7.2.15-0-VC15-installer.exe
Download Magento 2. You will need to register on the website first: magento.com/tech-resources/download
Create a new “magento” subfolder (or whatever name you prefer) inside the htdocs folder in the XAMPP installation (usually C:\xampp) and uncompress the Magento 2 archive there.
Start the XAMPP Control Panel from the Windows start menu. In the “Apache” section, click the “Config” button and, on the menu that appears, select “PHP (php.ini)”. Remove the semicolon before the extension=intl, extension=soap, and extension=xsl texts to enable the intl, soap and xsl extensions.

Screenshot of the XAMPP Control Panel with the PHP config menu displayed.

MySQL and Magento setup

We have all the files in place and the environment ready to start configuring the database and install Magento 2.

Start the Apache and MySQL services from the XAMPP Control Panel. Wait for the green status texts to appear.
Create a “magento” (or whatever name you prefer) database in MySQL from phpMyAdmin, installed already on XAMPP: localhost/phpmyadmin
Run the Magento 2 setup from localhost/magento (replace the “magento” part of the URL with whatever name you have chosen to host Magento). If the setup program requires to do any additional configuration change, do it as instructed. Do a screenshot or save the final page contents for later use.

Example of the page that will be displayed when the Magento 2 installation finishes.

Fixing known issues

When we finish the installation process, we will have a Magento 2 instance running on our host. But we’re not ready yet! There are a couple known bugs with Magento 2 and XAMPP at the moment I’m writing this post:

The Magento admin page may not work (it shows a blank page): Fix this issue by updating the “isPathInDirectories” function inside the Validator.php file as instructed in this article.
We might not be able to upload images (like a custom logo or product pictures): Fix this issue by updating the design_config_form.xml file as shown here.

After we finish making these changes, we will need to restart the Apache service from the XAMPP control panel. And that’s it! We should be ready to open the Magento front page, and login into the backend.

An empty home page with the default theme enabled in Magento 2.

This is how the Magento 2 back-end dashboard looks like once we login.

Setting up the store: A roadmap

The purpose of this article is to get Magento 2 up and running on Windows computer, and that’s what I hope we’ve achieved so far. But just to point in a direction on what to do next, below is my usual roadmap to set up the store, add a few products and publish the catalog on the home page.

Change the default store name and logo from the section Content > Design > Configuration and choose “Edit” in the “Default store view”.
Set up the default store configuration from the section Stores > Configuration.
Create the product categories from the section Catalog > Categories. Set up a category name and its dependency from another category, or from the default master category.
Create/edit the product attributes from the section Stores > Attributes (check the option “visible in storefront” to show the products in the product page). For example, custom attributes for book products would be Author, Publisher…
Add the attributes to the default attribute set from the section Stores > Attribute. This is required for the attributes to show up when adding a new product by default.
Create a set of products from the section Catalog > Products. Set the custom variations for each product from the “Customizable options” section. Set up a short and long description, a custom SKU for each customizable option, and a tax mode.
Add a widget to the homepage to show the product catalog from the section Content > Homepage > Edit > Show/Hide Editor > Add widget > Products list. Here we can also add any content we like, including HTML tags, images, and third-party content.

Conclusion

It’s possible—and pretty straightforward—to run a Magento 2 instance in Windows with XAMPP. But this is just a kick-off, a simple overview: there’s a lot more to do if we want to get a full ecommerce website that is ready to scale up and support enterprise-level traffic, like caching optimization, load balancing, and advanced server-side monitoring.

At End Point, we have professionals with a strong background in deploying powerful, reliable, fully responsive, fast, and SEO-optimized ecommerce websites. We have experience in migrating Magento to newer versions, and even creating full Magento 2 websites from scratch. Drop us a line if you want to hear more.

mysqldump issues after Percona 5.7 update

2017-04-07T00:00:00+00:00

During a recent CentOS 7 update, among other packages, we updated our Percona 5.7 installation to version 5.7.17-13.

Quickly after that, we discovered that mysqldump stopped working, thus breaking our local mysql backup script (that complained loudly).

What happened?

The error we received was:

mysqldump: Couldn't execute 'SELECT COUNT(*) FROM INFORMATION_SCHEMA.SESSION_VARIABLES WHERE VARIABLE_NAME LIKE 'rocksdb\_skip\_fill\_cache'': The 'INFORMATION_SCHEMA.SESSION_VARIABLES' feature is disabled; see the documentation for 'show_compatibility_56' (3167)

After a bit of investigation, we discovered this was caused by this regression bug, apparently already fixed but not yet available on CentOS:

Everything revolves around INFORMATION_SCHEMA being deprecated in version 5.7.6, when Performance Schema tables has been added as a replacement.

Basically, a regression caused mysqldump to try and use deprecated INFORMATION_SCHEMA tables instead of the new Performance Schema.

How to fix it?

Immediate workaround is to add this line to /etc/my.cnf or (more likely) /etc/percona-server.conf.d/mysqld.cnf, depending on how your configuration files are organized:

show_compatibility_56=1

This flag was both introduced and deprecated in 5.7.6. It will be there for some time to help with the transition.

It seems safe and, probably, good to keep if you have anything still actively using INFORMATION_SCHEMA tables, that would obviously be broken if not updated to the new Performance Schema since 5.7.6.

With this flag, it is possible to preserve the old behavior and keep your old code in a working state, while you upgrade it. Also, according to the documentation, it should not impact or turn off the new behavior with Performance Schema.

More information on how to migrate to the new Performance Schema can be found here.

Making cross-blogs queries in multi-site WordPress performant

2016-11-02T00:00:00+00:00

Some time ago I was working on customizing a WordPress system for a client. The system was running in a multi-site mode, being a host of a large number of blogs.

Because some blogs had not been updated in a long while, we wanted to pull information about recent posts from all of the blogs. This in turn was going to be used for pruning any blogs that weren’t considered “active”.

While the above description may sound simple, the scale of the system made the task a bit more involving that it would be usually.

How WordPress handles the “multi-site” scenario

The goal of computing the summary of posts for many blogs residing in the hypotethical blogging platform, in the same database doesn’t seem so complicated. Tasks like that are being performed all the time using relational databases.

The problem in WordPress arises though because of the very unusual way that it organises blogs data. Let’s see how the database tables look like in the “normal” mode first:

It has a number of tables that start with user configurable prefix. In the case of the screenshot above, the prefix was wp_.

We can see there’s a wp_posts table which contains rows related to blog posts. Thinking about the multi-blog setup, one would expect some kind of a blog_id column in the wp_posts column. Selecting data for the given blog would still be a cinch. It would also be very performant after adding an index on that column.

Instead, this is what we’re getting when setting up WordPress in a multi-site mode:

WordPress just creates a new set of tables we seen before appending the index of the blog to the tables prefix! Instead of having a nice, clean and easy to use wp_posts with the blog_id column, we get a number of tables—one for each blog: wp_1_posts, wp_2_posts etc. Why does it matter that much? Just try to get the counts of posts in each blog in one query—it’s impossible with such a tables setup. Getting such info involves querying each table separately. This means that with each new blog within the system, the cost of running such sequence of queries adds up dramatically. This is also known as a N+1 problem—bad WordPress! bad!

The approach around it

The counts of posts for each blog was needed to be computed very quickly in my case. The system consisted of hundreds of different mini-blogs and the stats were to be shown in the admin dashboard. Obviously making admins wait for the page to load for a long time wasn’t an option.

I went the route of creating an additional database table, holding the info about the number of posts for each blog. This table was then being updated upon each post creation, update and deletion. It was also updated upon the blog removal.

WordPress has a helper function for ensuring the table is created in the database. You feed it the DDL containing the definition of the table and it makes sure it is present in the database.

I created a function that was being fired on each plugin class instantiation, while making that class a singleton. Here’s the code that makes the plugin class a singleton:

class BlogsPruner
{
  private static $instance;

  private function __construct()
  {
    $this->ensureDatabaseSetup();
    $this->ensureBlogStatsGetUpdated();
    // ... other initialization here
  }

  public static function singleton()
  {
    if(!isset(self::$instance))
    {
      $_class = __CLASS__;
      self::$instance = new $_class;
    }
    return self::$instance;
  }

  public function __clone()
  {
    trigger_error('Cannot clone the pruner plugin', E_USER_ERROR);
  }

  // ... rest of the class
}

BlogsPruner::singleton();

The next step was to implement the function ensuring there’s a stats table in the database:

function ensureDatabaseSetup()
  {
    global $wpdb;
    $tableName = $this->blogStatsTableName();
    $charsetCollate = $wpdb->get_charset_collate();

    $sql = "CREATE TABLE $tableName (
            blog_id bigint(20),
            count_posts int(2),
            UNIQUE KEY blog_id (blog_id)
    ) $charsetCollate;";

    require_once( ABSPATH . 'wp-admin/includes/upgrade.php' );
    dbDelta( $sql );
    $this->ensureBlogStatsUpdated();
  }

This code uses a helper function to correctly construct the name for the table:

function blogStatsTableName()
{
  global $wpdb;
  return $wpdb->base_prefix . 'blog_stats';
}

This made sure the table was using the correct prefix, just like all the other tables in the database.

Now, I needed to ensure the stats were updated upon each post change:

function ensureBlogStatsGetUpdated()
{
  add_action('save_post', array($this, 'onPostUpdated'));
  add_action('delete_post', array($this, 'onPostDeleted'));
}

function onPostUpdated($postId)
{
  global $blog_id;
  $post = get_post($postId);
  if(wp_is_post_revision($postId) || $post->post_status == 'auto-draft')
  {
    return;
  }
  $this->updateBlogStats($blog_id);
}

function onPostDeleted()
{
  global $blog_id;
  $this->updateBlogStats($blog_id);
}

function updateBlogStats($blogId)
{
  $count = $this->getBlogUserCreatedPostsCount($blogId);
  $this->updateBlogPostsCount($blogId, $count);
}

// Here we're specifically not including the post that is auto-created
// upon the blog creation:
function getBlogUserCreatedPostsCount($blogId)
{
  global $wpdb;
  $sql = "SELECT
            COUNT(DISTINCT `wp_" . $blogId . "_posts`.`id`) AS count_user_posts
          FROM `wp_blogs`
          INNER JOIN `wp_" . $blogId . "_posts`
                  ON `wp_blogs`.`blog_id` = $blogId
          WHERE
            `wp_" . $blogId . "_posts`.`post_type` = 'post' AND
            `wp_" . $blogId . "_posts`.`post_status` = 'publish' AND
            TIMESTAMPDIFF(SECOND, `wp_" . $blogId . "_posts`.`post_date`, `wp_blogs`.`last_updated`) > 60";
  $row = $wpdb->get_row($sql);
  return intval($row->count_user_posts);
}

function updateBlogPostsCount($blogId, $count)
{
  global $wpdb;
  $data = array('count_posts' => $count);
  $where = array('blog_id' => $blogId);
  $wpdb->update($this->blogStatsTableName(), $data, $where);
}

The actual production plugin implemented many more features than this sample code demonstrates. It was listing the blogs that could be considered stale, automatically pruning them after specified in the admin screen time and allowing admins to configure it via the WordPress interface. The full set of features is beyond the scope of this post.

The overall result made getting the statistics about very large set of blogs very fast. The cost of querying for the number of posts of each blog was moved to incremental, small updates upon each post being created, modified or removed. For the end user, this cost was imperceptable.

Final thoughts

WordPress is loved by many users. If you’re not just a user but also working with the code, there’s a number of traps you may fall into though. If I were to employ techniques that get advised as “WordPress usual/default/preferred”—I’d end up with a very unhappy client who’s be owning a very broken WordPress system. Fortunately, the set of WordPress tables isn’t casted in stone and you can freely extend it—as long as you’re cautious and know what you’re doing. Provided that these two prerequisites are met—WordPress is just a database backed platform—like any other.

MySQL to PostgreSQL Migration Tips

2014-11-26T00:00:00+00:00

I recently was involved in a project to migrate a client’s existing application from MySQL to PostgreSQL, and I wanted to record some of my experiences in doing so in the hopes they would be useful for others.

Note that these issues should not be considered exhaustive, but were taken from my notes of issues encountered and/or things that we had to take into consideration in this migration process.

Convert the schema

The first step is to convert the equivalent schema in your PostgreSQL system, generated from the original MySQL.

We used mysqldump --compatible=postgresql --no-data to get a dump which matched PostgreSQL’s quoting rules. This file still required some manual editing to cleanup some of the issues, such as removing MySQL’s “Engine” specification after a CREATE TABLE statement, but this resulted in a script in which we were able to create a skeleton PostgreSQL database with the correct database objects, names, types, etc.

Some of the considerations here include the database collations/charset. MySQL supports multiple collations/charset per database; in this case we ended up storing everything in UTF-8, which matched the encoding of the PostgreSQL database, so there were no additional changes needed here; otherwise, it would have been necessary to note the original encoding of the individual tables and later convert that to UTF-8 in the next step.

We needed to make the following modifications for datatypes:

MySQL Datatype	PostgreSQL Datatype
tinyint	int
int(NN)	int
blob	bytea*
datetime	timestamp with timezone
int unsigned	int**
enum('1')	bool
longtext	text
varbinary(NN)	bytea

* Note: we ended up converting these specific fields to text, just given the data that was stored in these fields in actuality, which just goes to show you should review your data.

** Note: because PostgreSQL does not have unsigned numeric types, if this feature is an important part of your data model you can/should add a CHECK constraint to the column in question to check that the value is non-negative.

A few other syntactic changes; MySQL’s UNIQUE KEY in the CREATE TABLE statement needs to just be UNIQUE.

Some of the MySQL indexes were defined as FULLTEXT indexes as well, which was a keyword PostgreSQL did not recognize. We made note of these, then created just normal indexes for the time being, intending to review to what extent these actually needed full text search capabilities.

Some of the AUTO_INCREMENT fields did not get the DEFAULT value set correctly to a sequence, because those types just ended up as integers without being declared a serial field, so we used the following query to correct this:

-- cleanup missing autoincrement fields

WITH
datasource AS (
    SELECT
        k.table_name,
        k.column_name,
        atttypid::regtype,
        adsrc
    FROM
        information_schema.key_column_usage k
    JOIN
        pg_attribute
    ON
        attrelid = k.table_name :: regclass AND
        attname = k.column_name
    LEFT JOIN
        pg_attrdef
    ON
        adrelid = k.table_name :: regclass AND
        adnum = k.ordinal_position
    WHERE
        table_name IN (
            SELECT table_name::text FROM information_schema.key_column_usage WHERE constraint_name LIKE '%_pkey' GROUP BY table_name HAVING count(table_name) = 1
        ) AND
        adsrc IS NULL AND
        atttypid = 'integer' ::regtype
),
frags AS (
    SELECT
        quote_ident(table_name || '_' || column_name || '_seq') AS q_seqname,
        quote_ident(table_name) as q_table,
        quote_ident(column_name) as q_col
    FROM
        datasource
),
queries AS (
    SELECT
        'CREATE SEQUENCE ' || q_seqname || ';
' ||
        'ALTER TABLE ' || q_table || ' ALTER COLUMN ' || q_col || $$ SET DEFAULT nextval('$$ || q_seqname || $$');
    $$ ||
        $$SELECT setval('$$ || q_seqname || $$',(SELECT max($$ || q_col || ') FROM ' || q_table || '));
' AS query
    FROM frags
)
SELECT
    COALESCE(string_agg(query, E'\n'),$$SELECT 'No autoincrement fixes needed';$$) AS queries FROM queries
\gset

BEGIN;
:queries
COMMIT;

Basically the idea is that we look for all table with a defined integer primary key (hand-waving it it by using the _pkey suffix in the constraint name), but without a current default value, then generate the equivalent SQL to create a sequence and set that table’s default value to the nextval() for the sequence in question. We also generate SQL to scan that table and set that sequence value to the next appropriate value for the column in question. (Since this is for a migration and we know we’ll be the only user accessing these tables we can ignore MVCC.)

Another interesting thing about this script is that we utilize psql’s ability to store results in a variable, using the \gset command, then we subsequently execute this SQL by interpolating that corresponding variable in the same script.

Convert the data

The next step was to prepare the data load from a MySQL data-only dump. Using a similar dump recipe as for the initial import, we used: mysqldump --compatible=postgresql --no-create-info --extended-insert > data.sql to save the data in a dump file so we could iteratively tweak our approach to cleaning up the MySQL data.

Using our dump file, we attempted a fresh load into the new PostgreSQL database. This failed initially due to multiple issues, including ones of invalid character encoding and stricter datatype interpretations in PostgreSQL.

What we ended up doing was to create a filter script to handle all of the “fixup” issues needed here. This involved decoding the data and reencoding to ensure we were using proper UTF8, performing some context-sensitive datatype conversions, etc.

Additional schema modifications

As we were already using a filter script to process the data dump, we decided to take the opportunity to fixup some warts in the current table definitions. This included some fields which were varchar, but should have actually been numeric or integer; as this was a non-trivial schema (100 tables) we were able to use PostgreSQL’s system views to identify a list of columns which should should be numeric and were currently not.

Since this was an ecommerce application, we identified columns that were likely candidates for data type reassignment based on field names *count, *qty, *price, *num.

Once we identified the fields in question, I wrote a script to generate the appropriate ALTER TABLE statements to first drop the default, change the column type, then set the new default. This was done via a mapping between table/column name and desired output type.

Convert the application

The final (and needless to say most involved step) was to convert the actual application itself to work with PostgreSQL. Despite the fact that these databases both speak SQL, we had to come up for solutions for the following issues:

Quotation styles

MySQL is more lax with its quoting styles, so some of this migration involved hunting down differences in quoting styles. The codebase contained lots of double-quoted string literals, which PostgreSQL interprets as identifiers, as well as the difference in quoting of column names (backticks for MySQL, double-quotes for PostgreSQL). These had to be identified wherever they appeared and fixed to use a consistent quoting style.

Specific unsupported syntax differences:

INSERT ON DUPLICATE KEY

MySQL supports the INSERT ON DUPLICATE KEY syntax. Modifying these queries involved creating a special UPSERT-style function to support the different options in use in the code base. We isolated and categorized the uses of INSERT ON DUPLICATE KEY UPDATE into several categories: those which did a straight record replace and those which did some sort of modification. I wrote a utility script (detailed later in this article) which served to replicate the logic needed to handle this as the application would expect.

Upcoming versions of PostgreSQL are likely to incorporate an INSERT … ON CONFLICT UPDATE/IGNORE syntax, which would produce a more direct method of handling migration of these sorts of queries.

INSERT IGNORE

MySQL’s INSERT … IGNORE syntax allows you to insert a row and effectively ignore a primary key violation, assuming that the rest of the row is valid. You can handle this case via creating a similar UPSERT function as in the previous point. Again, this case will be easily resolved if PostgreSQL adopts the INSERT … ON CONFLICT IGNORE syntax.

REPLACE INTO

MySQL’s REPLACE INTO syntax effectively does a DELETE followed by an INSERT; it basically ensures that a specific version of a row exists for the given primary key value. We handle this case by just modifying these queries to do an unconditional DELETE for the Primary Key in question followed by the corresponding INSERT. We ensure these are done within a single transaction so the result is atomic.

INTERVAL syntax

Date interval syntax can be slightly different in MySQL; intervals may be unquoted in MySQL, but must be quoted in PostgreSQL. This project necessitated hunting down several instances to add quoting of specific literal INTERVAL instances.

Function considerations

last_insert_id()

Many times when you insert a records into a MySQL table, later references to this are found using the last_insert_id() SQL function. These sorts of queries need to be modified to utilize the equivalent functionality using PostgreSQL sequences, such as the currval() function.

GROUP_CONCAT()

MySQL has the GROUP_CONCAT function, which serves as a string “join” of sorts. We emulate this behavior in PostgreSQL by using the string_agg aggregate function with the delimiter of choice.

CONCAT_WS() — expected to be but not an issue; Postgres has this function

PostgreSQL has included a CONCAT_WS() function since PostgreSQL 9.1, so this was not an issue with the specific migration, but could still be an issue if you are migrating to an older version of PostgreSQL.

str_to_date()

This function does not exist directly in PostgreSQL, but can be simulated using to_date(). Note however that the format string argument differs between MySQL and PostgreSQL’s versions.

date_format()

MySQL has a date_format() function which transforms a date type to a string with a given format option. PostgreSQL has similar functionality using the to_char() function; the main difference here lies in the format string specifier.

DateDiff()

DateDiff() does not exist in PostgreSQL, this is handled by transforming the function call to the equivalent date manipulation operators using the subtraction (-) operator.

rand() to random()

This is more-or-less a simple function rename, as the equivalent functionality for returning a random float between 0.0 <= x <= 1.0 exists in PostgreSQL and MySQL, it’s just what the function name itself is. The other difference is that MySQL supports a scale argument so the random number for rand(N) will be returned between 0.0 <= x <= N, whereas you’d have to scale the result in PostgreSQL yourself, via random() * N.

IF() to CASE WHEN ELSE

MySQL has an IF() function which returns the second argument in the case the first argument evaluates to true otherwise returns the third argument. This can be trivially converted from IF(expression1, arg2, arg3) to the equilvalent PostgreSQL syntax: CASE WHEN expression1 THEN arg2 ELSE arg3.

IFNULL() to COALESCE()

MySQL has a function IFNULL() which returns the first argument if it is not NULL, otherwise it returns the second argument. This can effectively be replaced by the PostgreSQL COALESCE() function, which serves the same purpose.

split_part()

MySQL has a built-in function called split_part() which allows you to access a specific index of an array delimited by a string. PostgreSQL also has this function, however the split_part() function in MySQL allows the index to be negative, in which case this returns the part from the right-hand side.

in MySQL:

split_part('a banana boat', ' ', -1) => 'boat'

in PostgreSQL:

split_part('a banana boat', ' ', -1) => // ERROR: field position must be greater than zero

I fixed this issue by creating a custom PL/pgSQL function to handle this case. (In my specific case, all of the negative indexes were -1; i.e., the last element in the array, so I created a function to return only the substring occurring after the last instance of the delimiter.)

Performance considerations

You may need to revisit COUNT(*) queries

MySQL MyISAM tables have a very fast COUNT() calculation, owing to queries taking a table lock (which means MySQL can cache the COUNT() result itself, since there can be no concurrent updates), while PostgreSQL utilizes MVCC in order to calculate table counts which necessitates a full table scan to see which rows are visible to the specific calling snapshot, so this assumption may need to be revisited in order to calculate an equivalent performant query.

GROUP BY vs DISTINCT ON

MySQL is much more (ahem) flexible when it comes to GROUP BY/aggregate queries, allowing some columns to be excluded in a GROUP BY or an aggregate function. Making the equivalent query in PostgreSQL involves transforming the query from SELECT … GROUP BY (cols) to a SELECT DISTINCT ON (cols) … and providing an explicit sort order for the rows.

More notes

Don’t be afraid to script things; in fact, I would go so far as to suggest that everything you do should be scripted. This process was complicated and there were lots of moving parts to ensure moved in tandem. There were changes being made on the site itself concurrently, so we were doing testing against a dump of the original database at a specific point-in-time. Having everything scripted ensured that this process was repeatable and testable, and that we could get to a specific point in the process without having to remember anything I’d done off-the-cuff.

In addition to scripting the actual SQL/migrations, I found it helpful to script the solutions to various classifications of problems. I wrote some scripts which I used to create some of the various scaffolding/boilerplate for the tables involved. This included a script which would create an UPSERT function for a specific table given the table name, which was used when replacing the INSERT ON DUPLICATE KEY UPDATE functions. This generated script could then be tailored to handle more complex logic beyond a simple UPDATE. (One instance here is an INSERT ON DUPLICATE KEY UPDATE which increased the count of a specific field in the table instead of replacing the value.)

#!/usr/bin/env perl
# -*-cperl-*-

use strict;
use warnings;

use Data::Dumper;

my $table = shift or die "Usage: $0 <table>\n";
my @cols = @ARGV;

my $dbh = DBI->connect(...);

my @raw_cols = @{ $dbh->column_info(undef, 'public', $table, '%')->fetchall_arrayref({}) };
my %raw_cols = map { $_->{COLUMN_NAME} => $_ } @raw_cols;

die "Can't find table $table\n" unless @raw_cols;

my @missing_cols = grep { ! defined $raw_cols{$_} } @cols;

die "Referenced non-existing columns: @missing_cols\n" if @missing_cols;

my %is_pk;

unless (@cols) {
    @cols = map { $_->{COLUMN_NAME} } @raw_cols;
}

my @pk_cols = $dbh->primary_key(undef, 'public', $table);

@is_pk{@pk_cols} = (1)x@pk_cols;

my @data_cols = grep { ! $is_pk{$_} } @cols;

die "Missing PK cols from column list!\n" unless @pk_cols == grep { $is_pk{$_} } @cols;
die "No data columns!\n" unless @data_cols;

print <<EOF
CREATE OR REPLACE FUNCTION
    upsert_$table (@{[
    join ', ' => map {
        "_$_ $raw_cols{ $_ }->{pg_type}"
    } @cols
]})
RETURNS void
LANGUAGE plpgsql
AS \$EOSQL\$
BEGIN
    LOOP
        UPDATE $table SET @{[
    join ', ' => map { "$_ = _$_" } @data_cols
]} WHERE @{[
    join ' AND ' => map { "$_ = _$_" } @pk_cols
]};
        IF FOUND THEN
            RETURN;
        END IF;
        BEGIN
            INSERT INTO $table (@{[join ',' => @cols]}) VALUES (@{[join ',' => map { "_$_" } @cols]});
            RETURN;
        EXCEPTION WHEN unique_violation THEN
            -- Do nothing, and loop to try the UPDATE again.
        END;
    END LOOP;
END
\$EOSQL\$
;
EOF
;

This script created an upsert function from a given table to update all columns by default, also allowing you to create one with a different number of columns upserted.

I also wrote scripts which could handle/validate some of the column datatype changes. Since there were large numbers of columns which were changed, often multiple in the same table, I was able to have this script create a single ALTER TABLE statement with multiple ALTER COLUMN TYPE USING clauses, plus be able to specify the actual method that these column changes were to take place. These included several different approaches, depending on the target data type, but generally were to solve cases where there were fairly legitimate data that was not picked up by PostgreSQL’s input parsers. These included how to interpret blank fields as integers (in some cases we wanted it to be 0, in others we wanted it to be NULL), weird numeric formatting (leaving off numbers before or after the decimal point), etc.

We had to fix up in several locations missing defaults for AUTO_INCREMENT columns. The tables were created with the proper datatype, however we had to find tables which matched a specific naming convention and create/associate a sequence/serial column, set the proper default here, etc. (This was detailed above.)

There was a fair amount of iteration and customization in this process, as there was a fair amount of data which was not of the expected format. The process was iterative, and generally involved attempting to alter the table from within a transaction and finding the next datum which the conversion to the expected type did not work. This would result in a modification of the USING clause of the ALTER TABLE ALTER COLUMN TYPE to accommodate some of the specific issues.

In several cases, there were only a couple records which had bad/bunko data, so I included explicit UPDATE statements to update those data values via primary key. While this felt a bit “impure”, it was a quick and preferred solution to the issue of a few specific records which did not fit general rules.

Version 5 of Bucardo database replication system

2014-06-23T00:00:00+00:00

Goat & Kid by Flickr user Bala Sivakumar

Bucardo 5, the next generation of the async multimaster replication system, has been released. This major release removes the previous two source database limitation, allowing you to have as many sources (aka masters) and as many targets (aka slaves) as you wish. Bucardo can also replicate to other targets, including MySQL, MariaDB, Oracle, SQLite, MongoDB, and Redis. Bucardo has been completely rewritten and is more powerful and efficient than the previous version, known as Bucardo 4. You can always find the latest version here.

This article will show a quick demonstration of Bucardo. Future posts will explore its capabilities further: for now, we will show how easy it is to get basic multimaster replication up and running.

For this demo, I used a quick and disposable server from Amazon Web Services (AWS, specifically a basic t1.micro server running Amazon Linux). If you want to follow along, it’s free and simple to create your own instance. Once it is created and you have SSH’ed in as the ec2-user account, we can start to install PostgreSQL and Bucardo.

# Always a good idea:
$ sudo yum update
# This also installs other postgresql packages:
$ sudo yum install postgresql-plperl
# Create a new Postgres cluster:
$ initdb btest

We cannot start Postgres up yet, as this distro uses both /var/run/postgresql and /tmp for its socket directory. Once we adjust the permissions of the first directory, we can start Postgres, and then create our first test database:

$ sudo chmod 777 /var/run/postgresql
$ pg_ctl -D btest -l logfile start
$ createdb shake1

Next, we need something to replicate! For a sample dataset, I like to use the open source Shakespeare project. It’s a small, free, simple schema that is easy to load. There’s a nice little project on github the contains a ready-made Postgres schema, so we will load that in to our new database:

$ sudo yum install git
$ git clone -q https://github.com/catherinedevlin/opensourceshakespeare.git
$ psql shake1 -q -f opensourceshakespeare/shakespeare.sql
# You can safely ignore the 'role does not exist' errors

We want to create duplicates of this database, to act as the other sources. In other words, other servers that have the identical data and can be written to. Simple enough:

$ createdb shake2 -T shake1
$ createdb shake3 -T shake1

Bucardo has some dependencies that need installing. You may have a different list depending on your distro: this is what Amazon Linux needed when I wrote this. (If you are lucky, your distro may have Bucardo already available, in which case many of the steps below can be replaced e.g. with “yum install bucardo”—just make sure it is using version 5 or better! (e.g. with yum info bucardo))

$ sudo yum install  perl-ExtUtils-MakeMaker  perl-DBD-Pg \
> perl-Encode-Locale  perl-Sys-Syslog perl-boolean \
> perl-Time-HiRes  perl-Test-Simple  perl-Pod-Parser
$ sudo yum install cpan
$ echo y | cpan DBIx::Safe

The Perl module DBIx::Safe was not available on this system’s yum, hence we needed to install it via CPAN. Once all of that is done, we are ready to install Bucardo. We’ll grab the official tarball, verify it, untar it, and run make install:

$ wget -nv http://bucardo.org/Bucardo.tar.gz
$ wget -nv http://bucardo.org/Bucardo.tar.gz.asc
$ gpg -q --keyserver pgp.mit.edu --recv-key 14964AC8
$ gpg --verify Bucardo.tar.gz.asc
$ tar xfz Bucardo.tar.gz
$ ln -s Bucardo-5.0.0 bucardo
$ cd bucardo
$ perl Makefile.PL
$ make
$ sudo make install

Let’s make some small adjustments via the bucardorc file (which sets some global information). Then we can run the “bucardo install”, which creates the main bucardo database, containing the information the Bucardo daemon will need:

$ mkdir pid
$ echo -e "piddir=pid\nlogdest=." > .bucardorc
$ bucardo install --batch --quiet
Creating superuser 'bucardo'

Now that Bucardo is installed and ready to go, let’s setup the replication. In this case, we are going to have three of our databases replicate to each other. We can do all this in only two commands:

$ bucardo add dbs s1,s2,s3 dbname=shake1,shake2,shake3
Added databases "s1","s2","s3"
$ bucardo add sync bard dbs=s1:source,s2:source,s3:source tables=all
Added sync "bard"
Created a new relgroup named "bard"
Created a new dbgroup named "bard"
  Added table "public.chapter"
  Added table "public.character"
  Added table "public.character_work"
  Added table "public.paragraph"
  Added table "public.wordform"
  Added table "public.work"

With the first command, we told Bucardo how to connect to three databases, and gave them names for Bucardo to refer to as (s1,s2,s3). You can also specify the port and host, but in this case the default values of 5432 and no host (Unix sockets) were sufficient.

The second command creates a named replication system, called a sync, and assigns it the name “bard”. It needs to know where and how to replicate, so we tell it to use the three databases s1,s2, and s3. Each of these is to act as a source, so we append that information as well. Finally, we need to know what to replicate. In this case, we simply want all tables (or to be more precise, all tables with a primary key or a unique index). Notice that Bucardo always puts databases and tables into named groups—in this case, it was done automatically, and the dbgroup and relgroup are simply named after the sync.

Let’s verify that replication is working, by checking that a changed row replicates to all systems involved in the sync:

$ bucardo start
$ psql shake1 -c \
> "update character set speechcount=123 where charname='Hamlet'"
UPDATE 1
$ for i in {1,2,3}; do psql shake$i -tc "select \
> current_database(), speechcount from character \
> where charname='Hamlet'"; done | grep s
 shake1       |     123
 shake2       |     123
 shake3       |     123

We can also take a peek in the Bucardo log file, “log.bucardo” to see the replication happening:

$ tail -2 log.bucardo
(25181) KID (bard) Delta count for s1.public."character": 1
(25181) KID (bard) Totals: deletes=2 inserts=2 conflicts=0

There are two deletes and inserts because the changed row was first deleted, and then inserted (via COPY, technically) to the other two databases. Next, let’s see how Bucardo handles a conflict. We will have the same row get changed on all the servers, which should lead to a conflict:

$ for i in {1,2,3}; do psql shake$i -tc \
> "update character set speechcount=$i$i$i \
> where charname='Hamlet'"; done
UPDATE 1
UPDATE 1
UPDATE 1

Looking in the logs shows that we did indeed have a conflict, and that it was resolved. The default conflict resolution declares the last database to be updated the winner. All three databases now have the same winning row:

$ tail log.bucardo
(25181) KID (bard) Delta count for s1.public."character": 1
(25181) KID (bard) Delta count for s2.public."character": 1
(25181) KID (bard) Delta count for s3.public."character": 1
(25181) KID (bard) Conflicts for public."character": 1
(25181) KID (bard) Conflicts have been resolved
(25181) KID (bard) Totals: deletes=2 inserts=2 conflicts=1

$ for i in {1,2,3}; do psql shake$i -tc \
> "select current_database(), speechcount \
> from character where charname='Hamlet'"; done | grep s
 shake1       |     333
 shake2       |     333
 shake3       |     333

Sometimes when I was developing this demo, Bucardo was so fast that conflicts did not happen. In other words, because the updates were sequential, there is a window in which Bucardo can replicate a change before the next update occurs. The “pause a sync” feature can be very handy for this, as well as other times in which you need a sync to temporarily stop running:

$ bucardo pause bard
Syncs paused: bard
$ psql shake1 -c "update character set speechcount=1234 where charname='Hamlet'"
UPDATE 1
$ psql shake2 -c "update character set speechcount=4321 where charname='Hamlet'"
UPDATE 1
$ bucardo resume bard
Syncs resumed: bard

$ tail log.bucardo
(27344) KID (bard) Delta count for s1.public."character": 1
(27344) KID (bard) Delta count for s2.public."character": 1
(27344) KID (bard) Conflicts for public."character": 1
(27344) KID (bard) Conflicts have been resolved
(27344) KID (bard) Totals: deletes=2 inserts=2 conflicts=1

There is a lot more to Bucardo 5 than what is shown here. Future posts will cover other things it can do, from replicating to non-Postgres systems such as Oracle, MySQL, or MongoDB, to using custom conflict resolution mechanisms, to transforming data on the fly while replicating. If you have any questions, use the comments below, or drop a line to the Bucardo mailing list at bucardo-general@bucardo.org.

This major release would not have been possible without the help of many people over the years who have contributed code, raised bugs, tested Bucardo out, and asked (and/or answered!) important questions. Please see the Changes file for a partial list. Thanks to all of you, and special thanks to Jon Jensen, who started this whole project, many moons ago.

Why Can’t I Edit this Database Table? Don’t Forget the Client!

2014-06-11T00:00:00+00:00

A client of mine recently informed me of an issue he’d been having for years, where he was unable to edit a specific table in his database. He uses Access to connect to a MySQL database via ODBC, and his database has a few dozen tables, all of which are editable except this one. He reports that, when trying to edit just this one table, putting the cursor into any of the fields and attempting to change any of the data is blocked. As he put it, “It’s like the keyboard won’t respond.”

We confirmed through conversation that the issue was not a MySQL permissions problem—not that I would have expected MySQL permissions to result in such client behavior. We also confirmed that, when using a different application to connect to MySQL with Perl’s DBI, the table was editable just as the rest of the database. At this point, I didn’t have any good suspects (as neither Access nor ODBC are my strong suit) and agreed to bring up the issue with the rest of the End Point engineering team.

After sending out a description of the problem, it wasn’t long before Josh Williams responded. He had seen this sort of behavior with Access before, where the client will lock out the table if the table does not have a unique key defined. Not surprisingly, it turned out this particular table’s implied primary key was in fact a non-unique index. I applied a primary key to the field, dropped the original index, and received confirmation from the client that the table was now editable like the rest of the database.

Setting up your database is a combination of server and client behaviors. While most of the focus goes into configuring the server, if you encounter unusual circumstances, don’t forget the possibility that a given client may also have requirements impacting actions normally confined to the realm of the server.

Sanity, thy name is not MySQL

2014-04-10T00:00:00+00:00

Probably old news, but I hit this MySQL oddity today after a long day dealing with unrelated crazy stuff and it just made me go cross-eyed:

CREATE TABLE foo (id integer, val enum('','1'));
INSERT INTO foo VALUES (1, '');
INSERT INTO foo VALUES (2, '1');
SELECT * FROM foo WHERE val = 1;

What row do you get? I’ll wait while you second- and third-guess yourself.

It turns out that the “enum” datatype in MySQL just translates to a set of unique integer values. In our case, that means:

’’ == 1
‘1’ == 2

So you get the row with (1,’’). Now, if that doesn’t confuse readers of your code, I don’t know what will.

MySQL, ASCII Null, and Data Migration

2014-02-26T00:00:00+00:00

Data migrations always have a wide range of challenges. I recently took on a request to determine the difficulty of converting an ecommerce shop’s MySQL 5.0 database to PostgreSQL 9.3, with the first (presumably “easier”) step being just getting the schema converted and data imported before tackling the more challenging aspect of doing a full assessment of the site’s query base to re-write the large number of custom queries that leverage MySQL-specific language elements into their PostgreSQL counterparts.

During the course of this first part, which had contained a number of difficulties I had anticipated, I hit one that I definitely had not anticipated:

ERROR:  value too long for type character varying(20)

Surely, the error message is absolutely clear, but how could this possibly be? The obvious answer—that the varchar definitions were different lengths between MySQL and PostgreSQL—was sadly quite wrong (which you knew, or I wouldn’t have written this).

After isolating out the row in question, the first clear distinction of the data in question was the presence of the ASCII null character in it:

'07989409006\007989409'

OK, sure, that’s weird (and I have a hunch why it’s there) but I’m still puzzled how the “too long” data could be in MySQL in the first place. Then, things become both weirder and more clear when I go to take a look at the record as it exists in the source database of the dump. The table in question has many columns, so naturally I look at the results with the left-justified output of \G. Without displaying the full record for a number of reasons, here’s what I saw for that field:

*************************** 1. row ***************************
xxxxx: 07989409006
1 row in set (0.00 sec)

Wow! OK, so in MySQL, even though the ASCII null is clearly maintained in the data (the row came from a mysqldump directly from that same database) somehow the data are truncated to the string before the ASCII null, and so it must be determining string length from the same process that is truncating. To “prove” this to myself, I added to the where clause on the above query “AND xxxxx = ‘07989409006’”, and sure enough the record was not found.

I discussed all of my findings with a colleague, who was equally bewildered and recommended I post something describing the issue. I agreed, and to make sure I was talking about an existing problem (this was MySQL 5.0 after all) I went to reproduce the issue on a more modern release (v 5.5 to be exact).

I created a test case, with a new table having a single field into which I would insert a too-long string that had an ASCII null at a point such that the string to that point was less than the field length. My testing seemed to indicate the problem had been fixed sometime between 5.0 and 5.5:

mysql> select version();
+-----------------+
| version()       |
+-----------------+
| 5.5.35-33.0-log |
+-----------------+
1 row in set (0.00 sec)

mysql> create table foo (foo_var varchar(5));
Query OK, 0 rows affected (0.60 sec)

mysql> insert into foo values ('123\0456');
Query OK, 1 row affected, 1 warning (0.93 sec)

mysql> select * from foo;
+---------+
| foo_var |
+---------+
| 123 4   |
+---------+
1 row in set (0.00 sec)

Now to run the exact same test on my MySQL 5.0 instance, prove the deviation in behavior, and we’ll be good to go:

mysql> select version();
+------------+
| version()  |
+------------+
| 5.0.95-log |
+------------+
1 row in set (0.00 sec)

mysql> create table foo (foo_var varchar(5));
Query OK, 0 rows affected (0.00 sec)

mysql> insert into foo values ('123\0456');
Query OK, 1 row affected, 1 warning (0.00 sec)

mysql> select * from foo;
+---------+
| foo_var |
+---------+
| 123 4   |
+---------+
1 row in set (0.00 sec)

So, my contrived test also worked on the original MySQL server that had dumped out the data that led to this problem in the first place. What was going on here?

By now, I’m pretty seriously flailing about in indignation. What was so special about the data from the dump, and its behavior in the original table, that I had failed to understand and reproduce? I begin devising tests that seemed too absurd to be believed, as though I had two apples that, if I repositioned them just right, would prove 1+1 is actually 3. Then, by a stroke of fortune, I hit on the missing clue. It was a bug, all right, but not anything like what I was thinking:

mysql> select version();
+------------+
| version()  |
+------------+
| 5.0.95-log |
+------------+
1 row in set (0.00 sec)

mysql> select * from foo;
+---------+
| foo_var |
+---------+
| 123 4   |
+---------+
1 row in set (0.00 sec)

mysql> select * from foo \G
*************************** 1. row ***************************
foo_var: 123
1 row in set (0.00 sec)

Eureka! The bug was an issue rendering the output using the \G query terminator, whether restricted just to ASCII null or perhaps to other backslash-escaped characters, I do not know. Finally, to confirm whether the issue was still an issue or fixed over the intervening versions, I ran my new test on 5.5:

mysql> select version();
+-----------------+
| version()       |
+-----------------+
| 5.5.35-33.0-log |
+-----------------+
1 row in set (0.00 sec)

ql> select * from foo;
+---------+
| foo_var |
+---------+
| 123 4   |
+---------+
1 row in set (0.00 sec)

mysql> select * from foo \G
*************************** 1. row ***************************
foo_var: 123 4
1 row in set (0.00 sec)

Now I had a clear explanation for the behavior of the data with ASCII nulls, but it threw me back to starting over on the original length problem. However, without the red herring of the “too long” data being truncated to explain the size problem, I looked more closely at the specific data causing the problem and realized that counting \ and 0 as literal characters, instead of the metadata represetnation of ASCII null, the width of the data ended up exceeding the field length. Testing this hypothesis was simple enough:

mysql> select length('07989409006\007989409') AS len;
+-----+
| len |
+-----+
|  20 |
+-----+
1 row in set (0.00 sec)

#####

postgres=# select length('07989409006\007989409') AS len;
 len
-----
  21
(1 row)

This answer leaves some confusion on a couple of fronts. Why doesn’t PostgreSQL properly treat the \0 representation of ASCII null? The answer to this is, itself, a mixed answer. PostgreSQL will treat a string as a literal unless explicitly told to do otherwise. In order to tell the database a string has escape sequences in it, you have to open the string with the “E” (or “e”) identifier. However, doing that, we still don’t match the behavior we see in MySQL:

postgres=# select length(E'07989409006\007989409') AS len;
 len
-----
  18
(1 row)

The escape sequence consumed two additional characters because PostgreSQL interpreted the escape as octal \007:

\o, \oo, \ooo (o = 0 - 7) -> octal byte value1

The other potentially confusing point, then, is why didn’t MySQL interpret the escape sequence as octal \007? The simple answer there is, it just doesn’t do that; MySQL makes a special case of recognizing the ASCII null character as \0:

MySQL recognizes the escape sequences shown in Table 9.1, “Special Character Escape Sequences”. For all other escape sequences, backslash is ignored.2

If you look at Table 9.1, you’ll see the list of escape sequences is rather short and does not include general handling for octal or hex numeric representations.

Copy Data Between MySQL Databases with Sequel Pro

2014-01-10T00:00:00+00:00

Sequel Pro

I often use Sequel Pro when I’m getting up to speed on the data model for a project or when I just want to debug in a more visual way than with the mysql command-line client. It’s a free OS X application that lets you inspect and manage MySQL databases. I also find it very useful for making small changes to the data while I develop and test web apps.

Quickly Copy Data Between Databases

I recently needed a way to copy a few dozen records from one camp to another. I tried using the “SELECT…INTO OUTFILE” method but ran into a permissions issue with that approach. Using mysqldump was another option but that seemed like overkill in this case—I only needed to copy a few records from a single table. At this point I found a really neat and helpful feature in Sequel Pro: Copy as SQL INSERT

I simply selected the records I wanted to copy and used the “Copy as SQL INSERT” feature. The SQL insert statement I needed was now copied to the system clipboard and easily copied over to the other camp and imported via the mysql command-line client.

Bundles

The Sequel Pro website describes Bundles which extend the functionality in various ways—including copying data as JSON. Very handy stuff. Many thanks to the developers of this fine software. If you’re on OS X, be sure to give it a try.

Increasing MySQL 5.5 max_connections on RHEL 5

2013-12-24T00:00:00+00:00

Busy database-backed websites often hit scalability limits in the database first. In tuning MySQL, one of the first things to look at is the max_connections parameter, which is often too low. (Of course another thing to look at is appropriate fragment caching in your app server, HTTP object caching in your web server, and a CDN in front of it all.)

When using MySQL 5.5 from Oracle’s RPMs through cPanel (MySQL55-server-5.5.32-1.cp1136) on RHEL 5.10 x86_64, there is an interesting problem if you try to increase the max_connections setting beyond 214 in /etc/my.cnf. It will silently be ignored, and the limit remains 214:

mysql> show variables like 'max_connections';
+-----------------+-------+
| Variable_name   | Value |
+-----------------+-------+
| max_connections | 214   |
+-----------------+-------+
1 row in set (0.00 sec)

The problem is that the maximum number of open files allowed is too small, by default 1024, to increase max_connections beyond 214.

There are plenty of online guides that explain how to handle this, including increasing the kernel fs.file-max setting, which may be necessary by editing /etc/sysctl.conf, in this example to double the default:

fs.file-max = 2459688

Then run sysctl -p to make the change take immediate effect. (It’ll remain after reboot too.)

There are also many guides that say you need to change /etc/security/limits.conf along these lines:

mysql           soft    nofile         4096
mysql           hard    nofile         4096

However, the /etc/security/limits.conf change does not actually work when mysqld is started via the init script in /etc/init.d/mysql or via service mysql restart.

With standard Red Hat mysql-server (5.1) package that provides /etc/init.d/mysqld (not /etc/init.d/mysql as the Oracle and Percona versions do), you could create a file /etc/sysconfig/mysqld containing ulimit -n 4096 and that setting will take effect for each restart of the MySQL daemon.

But the ulimit -n setting hacked into the init script or put into /etc/sysconfig/mysqld isn’t really needed after all, because you can simply set open_files_limit in /etc/my.cnf:

[mysqld]
open_files_limit = 8192
max_connections = 1000
# etc.

… and mysqld_safe will increase the ulimit on its own before invoking the actual mysqld daemon.

After service mysql restart you can verify the new open file limit in the running process, like this:

# cat /var/lib/mysql/*.pid
30697
# ps auxww | grep 30697
mysql    30697 97.8  9.8 6031872 1212224 pts/1 Sl   13:09   3:01 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --user=mysql --log-error=/var/lib/mysql/some.hostname.err --open-files-limit=8192 --pid-file=/var/lib/mysql/some.hostname.pid
# cat /proc/30697/limits
Limit                     Soft Limit           Hard Limit           Units
Max cpu time              unlimited            unlimited            seconds
Max file size             unlimited            unlimited            bytes
Max data size             unlimited            unlimited            bytes
Max stack size            10485760             unlimited            bytes
Max core file size        0                    unlimited            bytes
Max resident set          unlimited            unlimited            bytes
Max processes             96086                96086                processes
Max open files            8192                 8192                 files
Max locked memory         32768                32768                bytes
Max address space         unlimited            unlimited            bytes
Max file locks            unlimited            unlimited            locks
Max pending signals       96086                96086                signals
Max msgqueue size         819200               819200               bytes
Max nice priority         0                    0
Max realtime priority     0                    0

And the running MySQL server will reveal the desired max_connections setting stuck this time:

mysql> show variables like 'max_connections';
+-----------------+-------+
| Variable_name   | Value |
+-----------------+-------+
| max_connections | 1000  |
+-----------------+-------+
1 row in set (0.00 sec)

The relevant code in /usr/bin/mysqld_safe is here:

if test -w / -o "$USER" = "root"
then
  # ... [snip] ...
  if test -n "$open_files"
  then
    ulimit -n $open_files
  fi
fi

if test -n "$open_files"
then
  append_arg_to_args "--open-files-limit=$open_files"
fi

I have found that some newer versions of either MySQL55-server or cPanel or some intersection of the two has made manually specifying a higher open_files_limit in /etc/my.cnf no longer necessary, although it does not do any harm.

But in conclusion, if you find yourself hitting the mysterious max_connections = 214 limit, just add the appropriately-sized open_files_limit to the [mysqld] section of /etc/my.cnf and restart the server with service mysql restart, and your problem should be solved!

How to Enable MySQL Event Scheduler

2013-11-26T00:00:00+00:00

You may think that you already know what’s the opposite of “DISABLED”, but with MySQL Event Scheduler you’ll be wrong.

In fact MySQL Event Scheduler may have three different states[1][2]:

DISABLED -  The Event Scheduler thread does not run [1]. In addition, the Event Scheduler state cannot be changed at runtime.
OFF (default) - The Event Scheduler thread does not run [1]. When the Event Scheduler is OFF it can be started by setting the value of event_scheduler to ON.
ON - The Event Scheduler is started; the event scheduler thread runs and executes all scheduled events.

So if you’re going to find it in the DISABLED state and instinctively set it to ENABLED you’ll end up with a non-starting MySQL daemon.

Be warned and stay safe out there!

[2]: When the Event Scheduler is not running does not appear in the output of SHOW PROCESSLIST

SFTP virtual users with ProFTPD and Rails: Part 2

2013-07-10T00:00:00+00:00

In Part 1 of “SFTP virtual users with ProFTPD and Rails”, I introduced ProFTPD’s virtual users and presented my annotated proftpd.conf that I used to integrate virtual users with a Rails application. Here in Part 2, I’ll show how we generate virtual user credentials, how we display them to the user, as well as our SftpUser ActiveRecord model that does the heavy lifting.

Let’s start at the top with the SFTP credentials UI. Our app’s main workflow actually has users doing most of their uploads through a sweet Plupload widget. So, by default, the SFTP functionality is hidden behind a simple button sitting to the right of the Plupload widget:

The user can click that button to open the SFTP UI, or the Plupload widget will open it automatically if the user tries to upload a file through the widget that is too big. Either way, it uses a jQuery UI function to slide the SFTP UI open. Before it makes the jQuery call, an Ajax request is made to request a new SFTP virtual user username and password. When that request returns, we populate those two textboxes. At that point, that virtual user exists as a new row in the sftp_users database table. At this point in the workflow, the user will be able to login using those credentials, and upload their files.

Let’s go look at our shiny new virtual user in the sftp_users table:

-[ RECORD 1 ]-+-----------------------------------
id            | 1
user_id       | 3
username      | user13A84C76
passwd        | {sha1}5kT0WDb/5H6C8M92dTeiKQO0Kg0=
uid           |
sftp_group_id |
homedir       | /home/sftp/uploads/3/user13A84C76
shell         |
created_at    | 2013-07-08 17:24:24.780753

The id and user_id fields are ignored by ProFTPD (they’re just standard primary/foreign key integers.) The username can be anything, so we just use “user” plus a random-enough hex string. The way we configured ProFTPD in Part 1 means our virtual users have absolutely nothing to do with normal system users, so the uid, sftp_group_id and shell fields can be empty.

The homedir has three components: “/home/sftp/uploads” must exist and be writable by the effective UID that is running your Rails app. The directory named “3” is the value from the user_id field. (Adding that additional level lets us associate these virtual users with actual web app users should we need to debug, etc.) Lastly, each virtual user gets its own directory underneath that. In this example, ProFTPD will create the “3” and “3/user13A84C76” directories when that virtual user logs in. The format of the value in the passwd field is highly dependent upon your proftpd.conf settings for SQLAuthTypes. The format shown here matches the proftpd.conf shown in Part 1 of this article, if you’re a copying and pasting type. The created_at field is ignored by ProFTPD, but it’s handy info to have.

Let’s look at the relevant methods implemented by our SftpUser ActiveRecord model.

  def hash_password(password)
    "{sha1}" + Base64.strict_encode64(Digest::SHA1.digest(generate_password))
  end

  def generate_username
    'user' + SecureRandom.hex(4).upcase
  end

  def generate_password
    SecureRandom.hex(8).downcase
  end

Admittedly, there’s no deep wizardry going on there. The critical piece is that the password gets hashed in the format that ProFTPD expects. Our hash_password method handles that for us here. (You can do whatever you want for the username and the raw password.) As a caveat, upgrading from Ruby 1.8.7 to 1.9.3 required replacing the Base64.encode64 call with Base64.strict_encode64.

Ok, that’s the end of Part 2. We now have a new virtual user with credentials that can be used to login via SFTP. We’ve displayed those credentials to the user using jQuery UI and Ajax. Each virtual user gets its own chrooted home directory that is created only if and when they log in. The next steps in the workflow will be discussed in Part 3: How the user signals they are done uploading via SFTP, using Resque to make the Rails app process the files uploaded by the user, preventing upload overlap and re-entrant problems, and cleaning up after a virtual user’s files are successfully processed offline by the Rails app.

Foreign Data Wrappers

2013-05-13T00:00:00+00:00

Original images from Flickr user jenniferwilliams

One of our clients, for various historical reasons, runs both MySQL and PostgreSQL to support their website. Information for user login lives in one database, but their customer activity lives in the other. The eventual plan is to consolidate these databases, but thus far, other concerns have been more pressing. So when they needed a report combining user account information and customer activity, the involvement of two separate databases became a significant complicating factor.

In similar situations in the past, using earlier versions of PostgreSQL, we’ve written scripts to pull data from MySQL and dump it into PostgreSQL. This works well enough, but we’ve updated PostgreSQL fairly recently, and can use the SQL/MED features added in version 9.1. SQL/MED (“MED” stands for “Management of External Data”) is a decade-old standard designed to allow databases to make external data sources, such as text files, web services, and even other databases look like normal database tables, and access them with the usual SQL commands. PostgreSQL has supported some of the SQL/MED standard since version 9.1, with a feature called Foreign Data Wrappers, and among other things, it means we can now access MySQL through PostgreSQL seamlessly.

The first step is to install the right software, called mysql_fdw. It comes to us via Dave Page, PostgreSQL core team member and contributor to many projects. It’s worth noting Dave’s warning that he considers this experimental code. For our purposes it works fine, but as will be seen in this post, we didn’t push it too hard. We opted to download the source and build it, but installing using pgxn works as well:

$ env USE_PGXS=1 pgxnclient install mysql_fdw
INFO: best version: mysql_fdw 1.0.1
INFO: saving /tmp/tmpjrznTj/mysql_fdw-1.0.1.zip
INFO: unpacking: /tmp/tmpjrznTj/mysql_fdw-1.0.1.zip
INFO: building extension
gcc -O2 -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -g -fpic -I/usr/include/mysql -I. -I. -I/home/josh/devel/pg91/include/postgresql/server -I/home/josh/devel/pg91/include/postgresql/internal -D_GNU_SOURCE -I/usr/include/libxml2   -c -o mysql_fdw.o mysql_fdw.c
mysql_fdw.c: In function ‘mysqlPlanForeignScan’:
mysql_fdw.c:466:8: warning: ‘rows’ may be used uninitialized in this function [-Wmaybe-uninitialized]
gcc -O2 -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -g -fpic -shared -o mysql_fdw.so mysql_fdw.o -L/home/josh/devel/pg91/lib -L/usr/lib  -Wl,--as-needed -Wl,-rpath,'/home/josh/devel/pg91/lib',--enable-new-dtags  -L/usr/lib/x86_64-linux-gnu -lmysqlclient -lpthread -lz -lm -lrt -ldl
INFO: installing extension
< ... snip ... >

Here I’ll refer to the documentation provided in mysql_fdw’s README. The first step in using a foreign data wrapper, once the software is installed, is to create the foreign server, and the user mapping. The foreign server tells PostgreSQL how to connect to MySQL, and the user mapping covers what credentials to use. This is an interesting detail; it means the foreign data wrapper system can authenticate with external data sources in different ways depending on the PostgreSQL user involved. You’ll note the pattern in creating these objects: each simply takes a series of options that can mean whatever the FDW needs them to mean. This allows the flexibility to support all sorts of different data sources with one interface.

The final step in setting things up is to create a foreign table. In MySQL’s case, this is sort of like a view, in that it creates a PostgreSQL table from the results of a MySQL query. For our purposes, we needed access to several thousand structurally identical MySQL tables (I mentioned the goal is to move off of this one day, right?), so I automated the creation of each table with a simple bash script, which I piped into psql:

for i in `cat mysql_tables`; do
    echo "CREATE FOREIGN TABLE mysql_schema.$i ( ... table definition ...)
        SERVER mysql_server OPTIONS (
            database 'mysqldb',
            query 'SELECT ... some fields ... FROM $i'
        );"
done

In a step not shown above, this script also consolidates the data from each table into one, native PostgreSQL table, to simplify later reporting. In our case, pulling the data once and reporting on the results is perfectly acceptable; in other words, data a few seconds old wasn’t a concern. We also didn’t need to write back to MySQL, which presumably could complicate things somewhat. We did, however, run into the same data validation problems PostgreSQL users habitually complain about when working with MySQL. Here’s an example, in my own test database:

mysql> create table bad_dates (mydate date);
Query OK, 0 rows affected (0.07 sec)

mysql> insert into bad_dates values ('2013-02-30'), ('0000-00-00');
Query OK, 2 rows affected (0.02 sec)
Records: 2  Duplicates: 0  Warnings: 0

Note that MySQL silently transformed ‘2013-02-30’ into ‘0000-00-00’. Sigh. Then, in psql we do this:

josh=# create extension mysql_fdw;
CREATE EXTENSION

josh=# create server mysql_svr foreign data wrapper mysql_fdw options (address '127.0.0.1', port '3306');
CREATE SERVER

josh=# create user mapping for public server mysql_svr options (username 'josh', password '');
CREATE USER MAPPING

josh=# create foreign table bad_dates (mydate date) server mysql_svr options (query 'select * from test.bad_dates');
CREATE FOREIGN TABLE

josh=# select * from bad_dates ;
ERROR:  date/time field value out of range: "0000-00-00"

We’ve told PostgreSQL we’ll be feeding it valid dates, but MySQL’s idea of a valid date differs from PostgreSQL’s, and the latter complains when the dates don’t meet its stricter requirements. Several different workarounds exist, including admitting that ‘0000-00-00’ really is wrong and cleaning up MySQL, but in this case, we modified the query underlying the foreign table to fix the dates on the fly:

SELECT CASE disabled WHEN '0000-00-00' THEN NULL ELSE disabled END,
    -- various other fields
    FROM some_table

Fortunately this is the only bit of MySQL / PostgreSQL impedance mismatch that has tripped us up thus far; we’d have to deal with any others we found individually, just as we did this one.

SFTP virtual users with ProFTPD and Rails: Part 1

2012-12-20T00:00:00+00:00

I recently worked on a Rails 3.2 project that used the sweet PLupload JavaScript/Flash upload tool to upload files to the web app. To make it easier for users to upload large and/or remote files to the app, we also wanted to let them upload via SFTP. The catch was, our users didn’t have SFTP accounts on our server and we didn’t want to get into the business of creating and managing SFTP accounts. Enter: ProFTPD and virtual users.

ProFTPD’s virtual users concept allows you to point ProFTPD at a SQL database for your user and group authentication. This means SFTP logins don’t need actual system logins (although you can mix and match if you want). Naturally, this is perfect for dynamically creating and destroying SFTP accounts. Give your web app the ability to create disposable SFTP credentials and automatically clean up after the user is done with them, and you have a self-maintaining system.

Starting from the inside-out, you need to configure ProFTPD to enable virtual users. Here are the relevant parts from our proftpd.conf:

##
# Begin proftpd.conf excerpt. For explanation of individual config directives, see the 
# great ProFTPD docs at http://www.proftpd.org/docs/directives/configuration_full.html
##
DefaultServer off
Umask 002
AllowOverwrite on

# Don't reference /etc/ftpusers
UseFtpUsers off

<ifmodule mod_sftp.c="">

# Enable SFTP
SFTPEngine on

# Enable SQL based authentication
SQLAuthenticate on

# From http://www.proftpd.org/docs/howto/CreateHome.html
# Note that the CreateHome params are kind of touchy and easy to break.
CreateHome on 770 dirmode 770 uid ~ gid ~

# chroot them to their home directory
DefaultRoot ~

# Defines the expected format of the passwd database field contents. Hint: An
# encrypted password will look something like: {sha1}IRYEEXBUxvtZSx3j8n7hJmYR7vg=
SQLAuthTypes OpenSSL

# That '*' makes that module authoritative and prevents proftpd from
# falling through to system logins, etc
AuthOrder mod_sql.c*

# sftp_users and sftp_groups are the database tables that must be defined with
# the proceeding column names. You can have other columns in these tables and
# ProFTPD will leave them alone. The sftp_groups table can be empty, but it must exist.
SQLUserInfo sftp_users username passwd uid sftp_group_id homedir shell
SQLGroupInfo sftp_groups name id members

SFTPHostKey /etc/ssh/ssh_host_rsa_key
SFTPHostKey /etc/ssh/ssh_host_dsa_key

SFTPCompression delayed
SFTPAuthMethods password
RequireValidShell no

# SQLLogFile is very verbose, but helpful for debugging while you're getting this working
SQLLogFile /var/log/proftpd_sql.sql

## Customize these for production
SQLConnectInfo database@localhost:5432 dbuser dbpassword

# The UID and GID values here are set to match the user that runs our web app because our
# web app needs to read and delete files uploaded via SFTP. Naturally, that is outside
# the requirements of a basic virtual user setup. But in our case, our web app user needs
# to be able to cd into a virtual user's homedir, and run a `ls` in there. Also, note that
# setting these two IDs here (instead of in our sftp_users table) *only* makes sense if
# you are using the DefaultRoot directive to chroot virtual users.
SQLDefaultUID  510
SQLDefaultGID  500

</ifmodule>

The CreateHome piece was the trickiest to get working just right for our use-case. But there are two reasons for that; we needed our web app to be able to read/delete the uploaded files, and we wanted to make ProFTPD create those home directories itself. (And it only creates that home directory once a user successfully logs in via SFTP. That means you can be more liberal in your UI with generating credentials that may never get used without having to worry about a ton of empty home directories lying about.)

That’s it for the introductory “Part 1” of this article. In Part 2, I’ll show how we generate credentials, the workflow behind displaying those credentials, and our SftpUser ActiveRecord model that handles it all. In Part 3, I’ll finish up by running through exactly how our web app accesses these files, and how it cleans up after it’s done.

Case Sensitive MySQL Searches

2012-10-18T00:00:00+00:00

MySQL’s support for case sensitive search is explained somewhat opaquely in the aptly titled Case Sensitivity in String Searches documentation. In short, it explains that by default, MySQL won’t treat strings as case sensitive when executing a statement such as:

SELECT first_name FROM contacts WHERE first_name REGEXP '^[a-z]';

This simple search to look for contacts whose first name starts with a lower case letter, will return all contacts because in the default character set used by MySQL (latin1), upper and lower case letters share the same “sort value”.

UPDATE: After many helpful comments from readers, it would seem the term I should have used was collation, not sort value. The documentation for both MySQL and PostgreSQL have lengthy discussions on the topic.

Enough with the backstory, how do I perform case sensitive searches!

The docs say to convert the string representation to a binary one. This allows “comparisons [to] use the numeric values of the bytes in the operands”. Let’s see it in action:

SELECT first_name FROM contacts WHERE BINARY(first_name) REGEXP '^[a-z]';

There are other strategies available, such as changing the character set being used for comparisons with the COLLATE function. This would likely work better for cases where you had many columns to compare.

SELECT first_name FROM contacts WHERE first_name REGEXP '^[a-z]' COLLATE latin1_bin;

You can even go so far as to have MySQL switch character sets and collations. But you do have to do this for each database, each table, and each column you need to convert. Not terribly fun.

Simple bash shell script for running batch MySQL jobs

2012-10-16T00:00:00+00:00

The other day I needed to run a simple mysql job to backup and delete some database records on a live server. Being a live server, it is important to make sure you aren’t asking the database to take on jobs that could potentially lock it up. Better to run a batch job. Running a batch is simple. You can call it right from the mysql console with:

source [path_to]/[the_batch_script].sql

But what if there are millions of records that need deleting? Bash shell script to the rescue.

Here is the idea of the SQL job that needed to get run a few times:

START TRANSACTION;

/* Find what you want to delete and put a LIMIT on your batch size */
CREATE TEMPORARY TABLE records_to_delete_temp SELECT id from `records` where ..... limit 1000;

/* Creating backup table to archive spam orders */
CREATE TABLE IF NOT EXISTS `records_backup` LIKE `records`;
INSERT INTO `records_backup` SELECT * from `records` where id in (select id from `records_to_delete_temp`);

/* Delete Dependents - If your records have foreign key dependencies, delete them first */
DELETE FROM `dependent_1` where record_id in (select id from `records_to_delete_temp`);
DELETE FROM `dependent_2` where record_id in (select id from `records_to_delete_temp`);

/* Delete the records */
DELETE FROM `records` where id in (select id from `records_to_delete_temp`);

/* Return count of remaining records - the where clause should be the same as the original select for the records to delete */
SELECT COUNT(*) from `records` where .... ;

COMMIT ;

Note:

SELECT COUNT(*)

This will return the remaining record count to the shell script.

And the shell script…

ret=$(mysql -u [user] -p[password] --skip-column-names [database_name]  < [the_batch_script].sql)

while [ $ret -gt 0 ]
do
  echo "rows left: $ret"
  sleep 3
  ret=$(mysql -u [user] -p[password] --skip-column-names [database_name]  < [the_batch_script].sql)
done

Notes:

--skip-column-names

This is the little nugget that gives you clean output from the mysql script. Without –skip-column-names you will have to get more creative with parsing the returned table.

-p[password]

Just a friendly reminder, no space after the -p option.

Should you really be running batches to remove millions of records from a live server with a shell script?

Just because you can doesn’t mean you should. Before pulling the trigger, back up, consider what could go wrong, have a plan in place to address the possible failure.

Find something fun for your hands to do while bash takes care of running the jobs. At the least, cross your fingers!

Points of Interest

2012-05-15T00:00:00+00:00

It’s been a fairly straight forward week at work, but I have stumbled a few interesting finds along the way this week.

Vim Adventures

Finally! A game based approach to learning Vim keyboard commands. I was hoping someone would do this. It’s just getting started (only two levels) and sadly, it looks like it’ll be charging money to unlock higher levels. However, some things are worth paying for. I’ve found just playing the first two levels a few times have helped retrain my brain to not take my fingers off the home row. It’s still quite buggy and seems to only work in Chrome. I found several times I needed to close all my Chrome windows after playing. Also, incognito mode seems to help with the bugs, as it disables all extensions you may have installed.

MySQL query comments in Rails

Ever wanted to know where that slow query was being called from? Well, if you’re using MySQL with your Rails 2.3.x or 3.x.x app, you can get debug information about what controller’s action made the call. Check out 37Signals new marginalia gem.

How to use EC2 as a web proxy

Kevin Burke provides a very detailed HOWTO article for working around restrictions you may experience in the course of an Internet based life. Pretty amazing what Amazon’s free usage tier puts out there; of course it’s only free for 12 months.

Include PIDs in your Logs

For many Rails developers we get comfortable looking at development log files. Sometimes when I have to investigate a customer issue on a production server using logs, I wished I had the level of detail the development logger had. While that’s a wish, I’m finding it mandatory to include PID numbers in my production logs. In production systems with multiple requests being handled simultaneously, Rails logs start to become unusable. It’s not clear which log lines are from which requests. Adding a PID in front of the time stamp can help untangle the mess. Here are some example approaches to this for Rails 3.x.x and Rails 2.3.x. Also, if you’re really a log-lover and manage a lot of servers, check out Papertrail, it looks very impressive for $7/mo.

Spectrum Shortages - Why it’s happening and what can be done

Telecom is not an area I have much familiarity with, but I found this article to be an interesting read. For example did you know that that largest owner of spectrum licenses are “under-capitalized or unwilling to build out networks” to use the spectrum? So while AT&T and Verizon struggle to meet the iPhone 4S’s data demands (twice as much as iPhone 4!), “there are some companies that have spectrum, but they’re struggling financially. Or they aren’t quite sure what to do with the spectrum. And others that have the money and business model, but need the spectrum.” It seems the way out of the mess is 4G, offering to improve the efficiency of spectrum use by 700 percent.

MySQL replication monitoring on Ubuntu 10.04 with Nagios and NRPE

2012-01-21T00:00:00+00:00

If you’re using MySQL replication, then you’re probably counting on it for some fairly important need. Monitoring via something like Nagios is generally considered a best practice. This article assumes you’ve already got your Nagios server setup and your intention is to add a Ubuntu 10.04 NRPE client. This article also assumes the Ubuntu 10.04 NRPE client is your MySQL replication master, not the slave. The OS of the slave does not matter.

Getting the Nagios NRPE client setup on Ubuntu 10.04

At first it wasn’t clear what packages would be appropriate packages to install. I was initially misled by the naming of the nrpe package, but I found the correct packages to be:

sudo apt-get install nagios-nrpe-server nagios-plugins

The NRPE configuration is stored in /etc/nagios/nrpe.cfg, while the plugins are installed in /usr/lib/nagios/plugins/ (or lib64). The installation of this package will also create a user nagios which does not have login permissions. After the packages are installed the first step is to make sure that /etc/nagios/nrpe.cfg has some basic configuration.

Make sure you note the server port (defaults to 5666) and open it on any firewalls you have running. (I got hung up because I forgot I have both a software and hardware firewall running!) Also make sure the server_address directive is commented out; you wouldn’t want to only listen locally in this situation. I recommend limiting incoming hosts by using your firewall of choice.

Choosing what NRPE commands you want to support

Further down in the configuration, you’ll see lines like command[check_users]=/usr/lib/nagios/plugins/check_users -w 5 -c 10. These are the commands you plan to offer the Nagios server to monitor. Review the contents of /usr/lib/nagios/plugins/ to see what’s available and feel free to add what you feel is appropriate. Well designed plugins should give you a usage if you execute them from the command line. Otherwise, you may need to open your favorite editor and dig in!

After verifying you’ve got your NRPE configuration completed and made sure to open the appropriate ports on your firewall(s), let’s restart the NRPE service:

service nagios-nrpe-server restart

This would also be an appropriate time to confirm that the nagios-nrpe-server service is configured to start on boot. I prefer the chkconfig package to help with this task, so if you don’t already have it installed:

sudo apt-get install chkconfig
chkconfig | grep nrpe

# You should see...
nagios-nrpe-server     on

# If you don't...
chkconfig nagios-nrpe-server on

Pre flight check - running check_nrpe

Before going any further, log into your Nagios server and run check_nrpe and make sure you can execute at least one of the commands you chose to support in nrpe.cfg. This way, if there are any issues, it is obvious now, while we’ve not started modifying your Nagios server configuration. The location of your check_nrpe binary may vary, but the syntax is the same:

check_nrpe -H host_of_new_nrpe_client -c command_name

If your command output something useful and expected, your on the right track. A common error you might see: Connection refused by host. Here’s a quick checklist:

Did you start the nagios-nrpe-server service?
Run netstat -lunt on the NRPE client to make sure the service is listening on the right address and ports.
Did you open the appropriate ports on all your firewall(s)?
Is there NAT translation which needs configuration?

Adding the check_mysql_replication plugin

There is a lot of noise out there on Google for Nagios plugins which offer MySQL replication monitoring. I wrote the following one using ideas pulled from several existing plugins. It is designed to run on the MySQL master server, check the master’s log position and then compare it to the slave’s log position. If there is a difference in position, the alert is considered Critical. Additionally, it checks the slave’s reported status, and if it is not “Waiting for master to send event”, the alert is also considered critical. You can find the source for the plugin at my Github account under the project check_mysql_replication. Pull that source down into your plugins directory (/usr/lib/nagios/plugins/ (or lib64)) and make sure the permissions match the other plugins.

With the plugin now in place, add a command to your nrpe.cfg.

command[check_mysql_replication]=sudo /usr/lib/nagios/plugins/check_mysql_replication.sh -H <slave_host_address></slave_host_address>

At this point you may be saying, WAIT! How will the user running this command (nagios) have login credentials to the MySQL server? Thankfully we can create a home directory for that nagios user, and add a .my.cnf configuration with the appropriate credentials.

usermod -d /home/nagios nagios #set home directory
mkdir /home/nagios
chmod 755 /home/nagios
chown nagios:nagios /home/nagios

# create /home/nagios/.my.cnf with your preferred editor with the following:
[client]
user=example_replication_username
password=replication_password

chmod 600 /home/nagios/.my.cnf
chown nagios:nagios /home/nagios/.my.cnf

This would again be an appropriate place to run a pre flight check and run the check_nrpe from your Nagios server to make sure this configuration works as expected. But first we need to add this command to the sudoer’s file.

nagios ALL= NOPASSWD: /usr/lib/nagios/plugins/check_mysql_replication.sh

Wrapping Up

At this point, you should run another check_nrpe command from your server and see the replication monitoring report. If not, go back and check these steps carefully. There are lots of gotchas and permissions and file ownership are easily overlooked. With this in place, just add the NRPE client using the existing templates you have for your Nagios servers and make sure the monitoring is reporting as expected.

MySQL Integer Size Attributes

2011-04-28T00:00:00+00:00

MySQL has those curious size attributes you can apply to integer data types. For example, when creating a table, you might see:

mysql> CREATE TABLE foo (
    -> field_ti tinyint(1),
    -> field_si smallint(2),
    -> field_int int(4),
    -> field_bi bigint(5)
    -> );
Query OK, 0 rows affected (0.05 sec)

mysql> desc foo;
+-----------+-------------+------+-----+---------+-------+
| Field     | Type        | Null | Key | Default | Extra |
+-----------+-------------+------+-----+---------+-------+
| field_ti  | tinyint(1)  | YES  |     | NULL    |       |
| field_si  | smallint(2) | YES  |     | NULL    |       |
| field_int | int(4)      | YES  |     | NULL    |       |
| field_bi  | bigint(5)   | YES  |     | NULL    |       |
+-----------+-------------+------+-----+---------+-------+
3 rows in set (0.03 sec)

mysql>

I had always assumed those size attributes were limiters, MySQL’s way of providing some sort of constraint on the integers allowed in the field. While doing some recent work for a MySQL client, I attempted to enforce the range of a tinyint according to that assumption. In reality, I only wanted a sign field, and would have liked to have applied a “CHECK field IN (-1,1)”, but without check constraints I figured at least keeping obviously incorrect data out would be better than nothing.

I wanted to see what MySQL’s behavior would be on data entry that failed the limiters. I was hoping for an error, but expecting truncation. What I discovered was neither.

mysql> INSERT INTO foo (field_ti) VALUES (-1);
Query OK, 1 row affected (0.00 sec)

mysql> SELECT field_ti FROM foo;
+----------+
| field_ti |
+----------+
|       -1 |
+----------+
1 row in set (0.00 sec)

mysql> INSERT INTO foo (field_ti) VALUES (1);
Query OK, 1 row affected (0.00 sec)

mysql> SELECT field_ti FROM foo;
+----------+
| field_ti |
+----------+
|       -1 |
|        1 |
+----------+
2 rows in set (0.00 sec)

mysql> INSERT INTO foo (field_ti) VALUES (10);
Query OK, 1 row affected (0.00 sec)

mysql> SELECT field_ti FROM foo;
+----------+
| field_ti |
+----------+
|       -1 |
|        1 |
|       10 |
+----------+
3 rows in set (0.00 sec)

mysql> INSERT INTO foo (field_ti) VALUES (100);
Query OK, 1 row affected (0.00 sec)

mysql> SELECT field_ti FROM foo;
+----------+
| field_ti |
+----------+
|       -1 |
|        1 |
|       10 |
|      100 |
+----------+
4 rows in set (0.00 sec)

mysql>

Two possible conclusions followed immediately: either the limiter feature was horribly broken, or those apparent sizes didn’t represent a limiter feature. A full review of MySQL’s Numeric Types documentation provided the answer:

MySQL supports an extension for optionally specifying the display width of integer data types in parentheses following the base keyword for the type. For example, INT(4) specifies an INT with a display width of four digits. This optional display width may be used by applications to display integer values having a width less than the width specified for the column by left-padding them with spaces. (That is, this width is present in the metadata returned with result sets. Whether it is used or not is up to the application.)

The display width does not constrain the range of values that can be stored in the column.

And, so, the lesson is repeated: Beware assumptions.

MySQL Ruby Gem CentOS RHEL 5 Installation Error Troubleshooting

2010-02-09T00:00:00+00:00

Building and installing the Ruby mysql gem on freshly-installed Red Hat based systems sometimes produces the frustratingly ambiguous error below:

# gem install mysql
/usr/bin/ruby extconf.rb
checking for mysql_ssl_set()... no
checking for rb_str_set_len()... no
checking for rb_thread_start_timer()... no
checking for mysql.h... no
checking for mysql/mysql.h... no
*** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of
necessary libraries and/or headers.  Check the mkmf.log file for more
details.  You may need configuration options.

Searching the web for info on this error yields two basic solutions:

Install the mysql-devel package (this provides the mysql.h file in /usr/include/mysql/).
Run gem install mysql – –with-mysql-config=/usr/bin/mysql_config or some other additional options.

These are correct but not sufficient. Because this gem compiles a library to interface with MySQL’s C API, the gcc and make packages are also required to create the build environment:

# yum install mysql-devel gcc make
# gem install mysql -- --with-mysql-config=/usr/bin/mysql_config

Alternatively, if you’re using your distro’s ruby (not a custom build like Ruby Enterprise Edition), you can install EPEL’s ruby-mysql package along with their rubygem-rails and other packages.

Gathering server information with boxinfo

2010-01-15T00:00:00+00:00

I’ve just publicly released another Postgres-related script, this one called “boxinfo”. Basically, it gathers information about a box (server), hence the catchy and original name. It outputs the information it finds into an HTML page, or into a MediaWiki formatted page.

The goal of boxinfo is to have a simple, single script that quickly gathers important information about a server into a web page, so that you can get a quick overview of what is installed on the server and how things are configured. It’s also useful as a reference page when you are trying to remember which server was it that had Bucardo version 4.5.0 installed and was running pgbouncer.

As we use MediaWiki internally here at End Point (running with a Postgres backend, naturally), the original (and default) format is HTML with some MediaWiki specific items inside of it.

Because it is meant to run on a wide a range of boxes as possible, it’s written in Perl. While we’ve run into a few boxes over the years that did not have Perl installed, the number that had any other language you choose (except perhaps sh) is much greater. It requires no other Perl modules, and simply makes a lot of system calls.

Various information about the box is gathered. System wide things such as mount points, disk space, schedulers, packaging systems are gathered first, along with versions of many common Unix utilities. We also gather information on some programs where more than just the version number is important, such as puppet, heartbeat, and lifekeeper. Of course, we also go into a great amount of detail about all the installed Postgres clusters on the box as well.

The program tries its best to locate every active Postgres cluster on the box, and then gathers information about it, such as where pg_xlog is linked to, any contrib modules installed, any interesting configuration variables from postgresql.conf, the size of each database, and lots of detailed information about any Slony or Bucardo configurations it finds.

The main page for it is on the Bucardo wiki at https://bucardo.org/wiki/Boxinfo. That page details the various command line options and should be considered the canonical documentation for the script. The latest version of boxinfo can be downloaded from that page as well. For any enhancement requests or problems to report, please submit an issue at https://github.com/bucardo/bucardo/issues.

What exactly does the output look like? We’ve got an example on the wiki showing the sample output from a run against my laptop. Some of the items were removed, but it should give you an idea of what the script can do, particularly with regards to the Postgres information: https://bucardo.org/Boxinfo/Example/

The script is still a little rough, so we welcome any patches, bug reports, requests, or comments. The development version can be obtained by running: git clone git://bucardo.org/boxinfo.git

State of the Postgres project

2010-01-04T00:00:00+00:00

It’s been interesting watching the MySQL drama unfold, but I have to take issue when people start trying to drag Postgres into it again by spreading FUD (Fear, Uncertainty, and Doubt). Rather than simply rebut the FUD, I thought this was a good opportunity to examine the strength of the Postgres project.

Monty recently espoused the following in a blog comment:

“…This case is about ensuring that Oracle doesn’t gain money and market share by killing an Open Source competitor. Today MySQL, tomorrow PostgreSQL.

Yes, PostgreSQL can also be killed; To prove the case, think what would happen if someone managed to ensure that the top 20 core PostgreSQL developers could not develop PostgreSQL anymore or if each of these developers would fork their own PostgreSQL project.”

Later on in his blog he raises the same theme again with a slight bit more detail:

“Note that not even PostgreSQL is safe from this threat! For example, Oracle could buy some companies developing PostgreSQL and target the core developers. Without the core developers working actively on PostgreSQL, the PostgreSQL project will be weakened tremendously and it could even die as a result.”

Is this a valid concern? It’s easy enough to overlook it considering the Sturm und Drang in Monty’s recent posts, but I think this is something worth seriously looking into. Specifically, is the Postgres project capable of withstanding a direct threat from a large company with deep pockets (e.g. Oracle)?

To get to the answer, let’s run some numbers first. Monty mentions the “top 20” Postgres developers. If we look at the community contributors page, we see that there are in fact 25 major developers listed, as well as 7 core members, so 20 would indeed be a significant chunk of that page. To dig deeper, I looked at the cvs logs for the year of 2009 for the Postgres project, and ran some scripts against them. The 9185 commits were spread across 16 different people, and about 16 other people were mentioned in the commit notes as having contributed in some way (e.g. a patch from a non-committer). So again, it looks like Monty’s number of 20 is a pretty good approximation.

However (and you knew there was a however), the catch comes from being able to actually stop 20 of those people from working on Postgres. There are basically two ways to do this: Oracle could buy out a company, or they could hire (buy out) a person. The first problem is that the Postgres community is very widely distributed. If you look at the people on the community contributors page, you’ll see that the 32 people work for 24 different companies. Further, no one company holds sway: the median is one company, and the high water mark is a mere three developers. All of this is much better than it was years ago, in the total number and in the distribution.

The next fly in the ointment is that buying out a company is not always easy to do, despite the size of your pockets. Many companies on that list are privately held and will not sell. Even if you did buy out the company, there is no way to prevent the people working there from then moving to a different company. Finally, buying out some companies just isn’t possible, even if you are Oracle, because there are some big names on the list of people employing major Postgres developers: Google, Red Hat, Skype, and SRA. Then of course there is NTT, which is a really, really big company (larger than Oracle). NTT’s Postgres developers are not always as visible as some of the English-speaking ones, but NTT employs a lot of people to work on Postgres (which is extremely popular in Japan).

The second way is hiring people directly. However, people can not always be bought off. Sure, some of the developers might choose to leave if Oracle offered them $20 million dollars, but not all of them (Larry, I might go for $19 million, call me :). Even if they did leave, the depth of the Postgres community should not be underestimated. For every “major developer” on that page, there are many others who read the lists, know the code well, but just haven’t, for one reason or another, made it on to that list. At a rough guess, I’d say that there are a couple hundred people in the world who would be able to make commits to the Postgres source code. Would all of them be as fast or effective as some of the existing people? Perhaps not, but the point is that it would be nigh impossible to thin the pool fast enough to make a dent.

The project’s email lists are as strong as ever, to such a point that I find it hard to keep up with the traffic, a problem I did not have a few years ago. The number of conferences and people attending each is growing rapidly, and there is a great demand for people with Postgres skills. The number of projects using Postgres, or offering it as an alternative database backend, is constantly growing. It’s no longer difficult to find a hosting provider that offers Postgres in addition to MySQL. Most important of all, the project continues to regularly release stable new versions. Version 8.5 will probably be released in 2010.

In conclusion, the state of the Postgres project is in great shape, due to the depth and breadth of the community (and the depth and breadth of the developer subset). There is no danger of Postgres going the MySQL route; the PG developers are spread across a number of businesses, the code (and documentation!) is BSD, and no one firm holds sway in the project.