Extensible Binary Encoding with CBOR

2019-03-18T00:00:00+00:00

CBOR is a relatively new IETF draft standard extensible binary data format. Compared to similar formats like MessagePack and BSON, CBOR was developed from the ground up with clear goals:

unambiguous encoding of most common data formats from Internet standards

code compactness for encoder and decoder

no schema description needed

reasonably compact serialization

applicability to constrained and unconstrained applications

good JSON conversion

extensibility

—RFC 7049 Appendix E, Copyright © 2013 IETF Trust, Bormann & Hoffman

In the context of data storage and messaging, most developers can relate to CBOR as a binary drop-in replacement for JSON. While CBOR doesn’t share the human readability of JSON, it can efficiently and unambiguously encode types of data that JSON struggles with. CBOR can also be extended with tags to optimize serialization beyond its standard primitives.

Encoding Binary Data

JSON is a ubiquitous data format for web and beyond, for many good reasons, but encoding blobs of binary data is an area where JSON falters. For example, if you are designing a JSON protocol to wrap the storage or transfer of arbitrary objects, your options are:

Require that all input data can be represented as JSON. When possible this is potentially a reasonable solution, but limits the types of data that can be encoded. Notable exceptions include most popular image encodings, excluding SVG.
Base64 encode any binary data values to a string. This can encode any binary data, but increases the size of the data by a minimum of 1/3, incurs encoding and decoding cost, and requires magic to indicate that the string is Base64 encoded.
Encode the bytes as an array of numbers or a hex string. These are probably not things you should do, but it seemed worth mentioning that these techniques increase the size of the data by anywhere from 2x to 5x and also require magic to indicate that the data is really binary.

With CBOR, binary blobs of any length are supported out of the box and are encoded 1:1. Encoding and decoding CBOR byte strings is extremely fast, even in higher-level languages. In the case of JavaScript, CBOR byte strings can be transcoded to and from the very fast and efficient Uint8Array.

For example, let’s try encoding a simple object with two fields: “name” and “data”. An abstract view of this object:

name: "Strawberry Pie"
data: <00 01 02 03 04 05 06 07 08 09>

A C struct of this raw data would be absolute minimum of 24 bytes.

If you Base64 the data into JSON, your output might look like this:

{"name":"Strawberry Pie","data":"AAECAwQFBgcICQ=="}

Length: 51 bytes.

This illustrates the “magic” problem with Base64 encoding binary data in JSON. Unless you have a JSON schema or special protocol fields, the decoder has no indication that the data in this object needs to be Base64 decoded, because it looks like a string. It is not self-describing.

The CBOR representation of the same input data, as a hex string:

a2646e616d656e5374726177626572727920506965696a7065675f646174614a00010203040506070809

Whoa, there. What do all these numbers and letters mean? I miss my JSON! The Node cbor package has a handy cbor2comment tool to annotate this hex string for us.

  a2                -- Map, 2 pairs
    64              -- String, length: 4
      6e616d65      -- {Key:0}, "name"
    6e              -- String, length: 14
      5374726177626572727920506965 -- {Val:0}, "Strawberry Pie"
    64              -- String, length: 4
      64617461 -- {Key:1}, "data"
    4a              -- Bytes, length: 10
      00010203040506070809 -- {Val:1}, 00010203040506070809

Length: 35 bytes.

Now let’s benchmark JSON against CBOR using real JPEG data. For this test I used a modified version of cbor-js, a library compatible with both Node and browsers, and encoded a 910,226 byte JPEG of strawberry rhubarb pie.

|                           | JSON      | CBOR    |
| :------------------------ | --------: | ------: |
| Median encoding time (ms) | 3.983     | 0.538   |
| Median decoding time (ms) | 3.151     | 0.006   |
| Encoded size (bytes)      | 1,213,676 | 910,262 |

As the numbers show, CBOR is both faster and more concise for this particular data. Also, CBOR pie tastes better.

Food always looks good in pictures!

Optimizing CBOR with Tags

In the case of encoding homogeneous numeric arrays, CBOR encoders can struggle with optimizing the packing of the data. For example, if you have an array of floating point numbers in a higher-level language like Python or JavaScript, the CBOR encoder implementation won’t necessarily determine how many bits are required to encode the numbers, defaulting to the largest available. Additionally, each value in the array will be individually described as a floating point number. This increases the cost and size of the data considerably.

CBOR has an answer to this problem. The Draft Typed Array Tags spec includes tags specifying typed arrays which happen to match JavaScript TypedArray flavors.

For example, let’s say you have a Float32Array with a few values:

const floats = new Float32Array([1.234567, 2.345678, 3.456789]);

JSON has a really funny way of encoding a Float32Array, lookit all those bytes:

{"0":1.2345670461654663,"1":2.3456780910491943,"2":3.456789016723633}

Length: 69 bytes.

If we were to help the encoder by sending it a regular Array it would still be pretty verbose, but the precision we weren’t using is truncated, so the overall length will vary wildly depending on the values in the Array:

[1.234567,2.345678,3.456789]

Length: 28 bytes.

The well-meaning Node cbor library can encode the Float32Array directly, but doesn’t try to optimize for size:

  83                -- Array, 3 items
    fb              -- Float, next 8 bytes
      3ff3c0c960000000 -- [0], 1.2345670461654663
    fb              -- Float, next 8 bytes
      4002c3f2e0000000 -- [1], 2.3456780910491943
    fb              -- Float, next 8 bytes
      400ba78100000000 -- [2], 3.456789016723633

Length: 28 bytes.

Look at all that wasted precision, 64 bits for each 32-bit float, this just won’t do! The CBOR spec allows you to “tag” data for special treatment. According to the list of registered CBOR tags, “IEEE 754 binary32, little endian, Typed Array” is tag 85. The consecutive bytes of the three numbers follow.

  d8                --  next 1 byte
    55              -- Tag #85
      4c            -- Bytes, length: 12
        4b069e3f971f1640083c5d40 -- 4b069e3f971f1640083c5d40

Length: 15 bytes.

The contained 12 byte string is equivalent to the values in the ArrayBuffer underneath our Float32Array.

The CBOR decoder will spit out the tag alongside its associated blob, so our optimization is self-described. Now we need to ensure that a byte string with this tag is correctly converted to the typed array, minding endianness. In a mature JavaScript implementation, this is both easy and very fast. Because we encoded the values of the ArrayBuffer underneath a Float32Array, we can construct a new Float32Array from the ArrayBuffer.

const floats = new Float32Array(bytes);  // assuming platform and bytes are same endianness!

This is a very scalable way to read and write arrays of numeric values into JavaScript.

When to Use CBOR

As King Crimson teaches us, “it doesn’t mean you should / just because you can.”

I’ve found CBOR useful as an alternative to JSON for large (>2kB) chunks of raw data or numeric arrays. In many other cases JSON is equivalent or superior, because most languages have native JSON encoding and decoding in their standard libraries. CBOR encoders are not always so optimized, but the gap is closing with wider adoption. Run your own tests and benchmarks on real data against real libraries before deciding to use CBOR.

Many readers will recognize that schemaful formats such as protocol buffers are an endgame for structured data. If the infrastructure demands make sense for your application, this is also a good way to go. If you’re working with an application that already uses JSON, the difference in development and maintenance costs of porting it to CBOR or protobuf-likes should be measured against the size and performance gains of each approach.

It’s tempting to let numbers alone decide what format to use, but don’t underestimate the value of JSON’s human readability for structured data. The value of human readability diminishes quickly when dealing with large numeric arrays or binary blobs, so once again CBOR is an appealing choice for these data.

wroc_love.rb 2017 part 2: The Elixir Hype

2017-03-21T00:00:00+00:00

One of the main reasons I attend wroc_love.rb almost every year, is that it’s a great forum for confronting ideas. It’s almost a tradition to have at least 2 very enjoyful discussion panels during this conference. One of them was devoted to Elixir and why the Ruby [1] community is so hyping about it.

Why Elixir is “sold” to us as “new better Ruby” while its underlying principles are totally different? Won’t it result in Elixir programmers that do not understand Elixir (like Rails programmers that do not know Ruby)?

Panelists discussed briefly the history of Elixir:

Jose Valim (who created Elixir) was working on threading in Rails and he was searching for better approaches for threading in web frameworks. He felt like lots of things were lacking in Erlang and Elixir is his approach for better Exceptions, better developer experience.

Then they jumped to Elixir’s main goals which are:

Compatibility with Erlang (all datatypes)
Better tooling
Improving developers’ experience

After that, they started speculating about problems that Elixir solves and RoR doesn’t:

Ruby on Rails addresses many problems in ways that may be somehow archaic to us in the ever-scaling world of 2017. There are many approaches to it, e.g. “actor model” which is implemented in Ruby by Celluloid, in Scala with Akka and also Elixir and Phoenix (Elixir’s rails-like framework) has its own actor model.

Phoenix (“Rails for Elixir”) is just an Elixir app—unlike Rails, it is not separate from Elixir. Moreover Elixir is exactly the same language as Erlang so:

Erlang = Elixir = Phoenix

Great comment:

Elixir is same as Erlang but not really #wrocloverb
— Daria Woznicka (@DWoznicka) March 18, 2017

Then a discussion followed during which panelists speculated about the price of the jump from Rails to Elixir:

The Java to Rails jump was caused by business/productivity. There’s no such jump for Phoenix/Elixir. Elixir code is more verbose (less instance variables, all args are passed explicitly to all functions).

My conclusions

A reason why this discussion was somehow shallow and pointless was that those two worlds have different problems. Great comment:

Ruby guys focus on business. Elixir guys on technical aspects #wrocloverb
— Michał Łomnicki (@mlomnicki) March 18, 2017

Elixir solves a lot of technical problems with scaling thanks to Erlang’s virtual machine. Such problems are definitely only a small part of what typical Ruby problem-solvers deal with on a daily basis. Hearing Elixir and Ruby on Rails developers talk past each other was probably the first sign of the fact that there’s no hyping technology right now. Each problem can be addressed by many tech tools and frameworks.

[1] Wrocloverb describes itself as a “best Java conference in Ruby world”. It’s deceiving:

#wrocloverb was great Clojure conference :)
Thanks for all organizers and speakers!
— Jakub Cieślar (@jcieslar_) March 19, 2017

Pgbouncer user and database pool_mode with Scaleway

2015-09-24T00:00:00+00:00

(Waterfall photo by Luke Price)

The recent release of PgBouncer 1.6, a connection pooler for Postgres, brought a number of new features. The two I want to demonstrate today are the per-database and per-use pool_modes. To get this effect previously, one had to run separate instances of PgBouncer. As we shall see, a single instance can now run different pool_modes seamlessly.

There are three pool modes available in PgBouncer, representing how aggressive the pooling becomes: session mode, transaction mode, and statement mode.

Session pool mode is the default, and simply allows you to avoid the startup costs for new connections. PgBouncer connects to Postgres, keeps the connection open, and hands it off to clients connecting to PgBouncer. This handoff is faster than connecting to Postgres itself, as no new backends need to be spawned. However, it offers no other benefits, and many clients/applications already do their own connection pooling, making this the least useful pool mode.

Transaction pool mode is probably the most useful one. It works by keeping a client attached to the same Postgres backend for the duration of a transaction only. In this way, many clients can share the same backend connection, making it possible for Postgres to handle a large number of clients with a small max_connections setting (each of which consumes resources).

Statement pool mode is the most interesting one, as it makes no promises at all to the client about maintaining the same Postgres backend. In other words, every time a client runs a SQL statement, PgBouncer releases that connections back to the pool. This can make for some great performance gains, although it has drawbacks, the primary one being no multi-statement transactions.

To demonstrate the new pool_mode features, I decided to try out a new service mentioned by a coworker, called Scaleway. Like Amazon Web Services (AWS), it offers quick-to-create cloud servers, ideal for testing and demonstrating. The unique things about Scaleway is the servers are all ARM-based SSDs. Mini-review of Scaleway: I liked it a lot. The interface was smooth and uncluttered (looking askance at you, AWS), the setup was very fast, and I had no problems with it being ARM.

To start a new server (once I entered my billing information, and pasted my public SSH key in), I simply clicked the create server button, chose “Debian Jessie (8.1)”, and then create server again. 60 seconds later, I had an IP address to login as root. The first order of business, as always, is to make sure things are up to date and install some important tools:

root@scw-56578065:~# apt-get update
root@scw-56578065:~# apt-get upgrade
## Only five packages were upgraded, which means things are already very up to date

## Because just plain 'apt-get' only gets you so far:
root@scw-56578065:~# apt-get install aptitude

## To find out the exact names of some critical packages:
root@scw-56578065:~# aptitude search emacs git

## Because a server without emacs is like a jungle without trees:
root@scw-56578065:~# apt-get install emacs-nox git-core

## To figure out what version of Postgres is available:
root@scw-56578065:~# aptitude show postgresql
Package: postgresql
State: not installed
Version: 9.4+165

## Since 9.4 is the latest, we will happily use it for this demo:
root@scw-56578065:~# apt-get install postgresql postgresql-contrib

## Nice to not have to worry about initdb anymore:
root@scw-56578065:~# service postgresql start

Postgres 9.4 is now installed, and started up. Time to figure out where the configuration files are and make a few small changes. We will turn on some heavy logging via the postgresql.conf file, and allow anyone locally to log in to Postgres, no questions asked, by changing the pg_hba.conf file. Then we restart Postgres, and verify it is working:

root@scw-56578065:~# updatedb
root@scw-56578065:~# locate postgresql.conf pg_hba.conf

root@scw-56578065:~# echo "logging_collector = on
log_filename = 'postgres-%Y-%m-%d.log'
log_rotation_size = 0" >> /etc/postgresql/9.4/main/postgresql.conf

## But it already has a nice log_line_prefix (bully for you, Debian)

## Take a second to squirrel away the old version before overwriting:
root@scw-56578065:~# cp /etc/postgresql/9.4/main/pg_hba.conf ~
root@scw-56578065:~# echo "local all all trust" > /etc/postgresql/9.4/main/pg_hba.conf
root@scw-56578065:~# service postgresql restart
root@scw-56578065:~# psql -U postgres -l
                             List of databases
   Name    |  Owner   | Encoding  | Collate | Ctype |   Access privileges
-----------+----------+-----------+---------+-------+-----------------------
 postgres  | postgres | SQL_ASCII | C       | C     |
 template0 | postgres | SQL_ASCII | C       | C     | =c/postgres          +
           |          |           |         |       | postgres=CTc/postgres
 template1 | postgres | SQL_ASCII | C       | C     | =c/postgres          +
           |          |           |         |       | postgres=CTc/postgres
(3 rows)

SQL_ASCII? Yuck, how did that get in there?! That’s an absolutely terrible encoding to be using in 2015, so we need to change that right away. Even though it won’t affect this demonstration, there is the principle of the matter. We will create a new database with a sane encoding, then create some test databases based on that.

root@scw-56578065:~# su - postgres
postgres@scw-56578065:~$ createdb -T template0 -E UTF8 -l en_US.utf8 foo
postgres@scw-56578065:~$ for i in {1..5}; do createdb -T foo test$i; done
postgres@scw-56578065:~$ psql -l
                             List of databases
   Name    |  Owner   | Encoding  |  Collate   |   Ctype    |   Access privileges
-----------+----------+-----------+------------+------------+-----------------------
 foo       | postgres | UTF8      | en_US.utf8 | en_US.utf8 |
 postgres  | postgres | SQL_ASCII | C          | C          |
 template0 | postgres | SQL_ASCII | C          | C          | =c/postgres          +
           |          |           |            |            | postgres=CTc/postgres
 template1 | postgres | SQL_ASCII | C          | C          | =c/postgres          +
           |          |           |            |            | postgres=CTc/postgres
 test1     | postgres | UTF8      | en_US.utf8 | en_US.utf8 |
 test2     | postgres | UTF8      | en_US.utf8 | en_US.utf8 |
 test3     | postgres | UTF8      | en_US.utf8 | en_US.utf8 |
 test4     | postgres | UTF8      | en_US.utf8 | en_US.utf8 |
 test5     | postgres | UTF8      | en_US.utf8 | en_US.utf8 |
(9 rows)

## Create some test users as well:
postgres@scw-56578065:~$ for u in {'alice','bob','eve','mallory'}; do createuser $u; done
## First time I tried this I outfoxed myself - so make sure the users can connect!
postgres@scw-56578065:~$ for d in {1..5}; do psql test$d -qc 'grant all on all tables in schema public to public'; done

## Make sure we can connect as one of our new users:
postgres@scw-56578065:~$ psql -U alice test1 -tc 'show effective_cache_size'
 847283

Now that Postgres is up and running, let’s install PgBouncer. Since we are showing off some 1.6 features, it is unlikely to be available via packaging, but we will check anyway.

postgres@scw-56578065:~$ aptitude versions pgbouncer
Package pgbouncer:
p   1.5.4-6+deb8u1            stable            500

## Not good enough! Let's grab 1.6.1 from git:
postgres@scw-56578065:~$ git clone https://github.com/pgbouncer/pgbouncer
postgres@scw-56578065:~$ cd pgbouncer
## This submodule business for such a small self-contained project really irks me :)
postgres@scw-56578065:~/pgbouncer$ git submodule update --init
postgres@scw-56578065:~/pgbouncer$ git checkout pgbouncer_1_6_1
postgres@scw-56578065:~/pgbouncer$ ./autogen.sh

The autogen.sh script fails rather quickly with an error about libtool—which is to be expected, as PgBouncer comes with a small list of required packages in order to build it. Because monkeying around with all those prerequisites can get tiresome, apt-get provides an option called “build-dep” that (in theory!) allows you to download everything needed to build a specific package. Before doing that, let’s drop back to root and give the postgres user full sudo permission, so we don’t have to keep jumping back and forth between accounts:

postgres@scw-56578065:~/pgbouncer$ exit
## This is a disposable test box - do not try this at home!
## Debian's /etc/sudoers has #includedir /etc/sudoers.d, so we can do this:
root@scw-56578065:~# echo "postgres ALL= NOPASSWD:ALL" > /etc/sudoers.d/postgres
root@scw-56578065:~# su - postgres
postgres@scw-56578065:~ cd postgres
postgres@scw-56578065:~/pgbouncer$ sudo apt-get build-dep pgbouncer
The following NEW packages will be installed:
  asciidoc autotools-dev binutils build-essential cdbs cpp cpp-4.9 debhelper docbook-xml docbook-xsl dpkg-dev g++ g++-4.9 gcc gcc-4.9 gettext
  gettext-base intltool-debian libasan1 libasprintf0c2 libatomic1 libc-dev-bin libc6-dev libcloog-isl4 libcroco3 libdpkg-perl libevent-core-2.0-5
  libevent-dev libevent-extra-2.0-5 libevent-openssl-2.0-5 libevent-pthreads-2.0-5 libgcc-4.9-dev libgomp1 libisl10 libmpc3 libmpfr4 libstdc++-4.9-dev
  libubsan0 libunistring0 libxml2-utils linux-libc-dev po-debconf sgml-data xmlto xsltproc
postgres@scw-56578065:~/pgbouncer$ ./autogen.sh

Whoops, another build failure. Well, build-dep isn’t perfect, turns out we still need a few packages. Let’s get this built, create some needed directories, tweak permissions, find the location of the installed PgBouncer ini file, and make a few changes to it:

postgres@scw-56578065:~/pgbouncer$ sudo apt-get install libtools automake pkg-config
postgres@scw-56578065:~/pgbouncer$ ./autogen.sh
postgres@scw-56578065:~/pgbouncer$ ./configure
postgres@scw-56578065:~/pgbouncer$ make
postgres@scw-56578065:~/pgbouncer$ sudo make install
postgres@scw-56578065:~/pgbouncer$ sudo updatedb
postgres@scw-56578065:~/pgbouncer$ locate pgbouncer.ini
/var/lib/postgresql/pgbouncer/etc/pgbouncer.ini
postgres@scw-56578065:~/pgbouncer$ sudo mkdir /var/log/pgbouncer /var/run/pgbouncer
postgres@scw-56578065:~/pgbouncer$ sudo chown postgres.postgres /var/log/pgbouncer \
 /var/run/pgbouncer /var/lib/postgresql/pgbouncer/etc/pgbouncer.ini
postgres@scw-56578065:~/pgbouncer$ emacs /var/lib/postgresql/pgbouncer/etc/pgbouncer.ini
## Add directly under the [databases] section:
test1 = dbname=test1 host=/var/run/postgresql
test2 = dbname=test2 host=/var/run/postgresql pool_mode=transaction
test3 = dbname=test3 host=/var/run/postgresql pool_mode=statement
test4  = dbname=test3 host=/var/run/postgresql pool_mode=statement auth_user=postgres
## Change listen_port to 5432
## Comment out listen_addr
## Make sure unix_socket_dir is /tmp

How are we able to use 5432, when Postgres is using it too? In the unix world, a port relies on a socket file, which is located somewhere on the file system. Thus, you can use the same port as long as they are coming from different files. While Postgres has a default unix_socket_directories value of ’/tmp’, Debian has changed that to ’/var/run/postgresql’, meaning PgBouncer itself is free to use ’/tmp’! The bottom line is that we can use port 5432 for both Postgres and PgBouncer, and control which one is used by setting the host parameter when connecting (which, when starting with a slash, is actually the location of the socket file). However, note that only one of them can be used when connecting via TCP/IP. Enough of all that, let’s make sure PgBouncer at least starts up!

postgres@scw-56578065:~/pgbouncer$ pgbouncer /var/lib/postgresql/pgbouncer/etc/pgbouncer.ini -d
2000-08-04 02:06:08.371 5555 LOG File descriptor limit: 65536 (H:65536),
 max_client_conn: 100, max fds possible: 230
postgres@scw-56578065:~/pgbouncer$

As expected, the pgbouncer program gave us a single line of information before going into background daemon mode, per the -d argument. Since both Postgres and PgBouncer are running on port 5432, let’s make our psql prompt a little more informative, by having it list the hostname via %M. If the hostname matches the unix_socket_directory value that psql was compiled with, then it will simply show ’[local]’. Thus, seeing ’/tmp’ indicates we are connected to PgBouncer, and seeing ’[local]’ indicates we are connected to Postgres (via /var/run/postgresql).

## Visit the <a href="http://www.postgresql.org/docs/current/static/app-psql.html#APP-PSQL-PROMPTING">psql docs</a> for explanation of this prompt
postgres@scw-56578065:~/pgbouncer$ echo "\set PROMPT1 '%n@%/:%>%R%x%#%M '" > ~/.psqlrc

Let’s confirm that each PgBouncer connection is in the expected mode. Database test1 should be using the default pool_mode, “session”. Database test2 should be using a “transaction” pool_mode, while “statement” mode should be used by both test3 and test4. See my previous blog post on ways to detect the various pool_modes of pgbouncer. First, let’s connect to normal Postgres and verify we are not connected to PgBouncer by trying to change to a non-existent database. FATAL means PgBouncer, and ERROR means Postgres:

postgres@scw-56578065:~/pgbouncer$ psql test1
psql (9.4.3)
Type "help" for help.

postgres@test1:5432=#[local] \c crowdiegocrow
FATAL:  database "crowdiegocrow" does not exist

postgres@scw-56578065:~/pgbouncer$ psql -h /tmp test1
psql (9.4.3)
Type "help" for help.

postgres@test1:5432=#[local:/tmp] \c sewdiegosew
ERROR:  No such database: sewdiegosew

Now let’s confirm that we have database-specific pool modes working. If you recall from above, test2 is set to transaction mode, and test3 is set to statement mode. We determine the mode by running three tests. First, we do a “BEGIN; ROLLBACK;”—if this fails, it means we are in statement mode. Next, we try to PREPARE and EXECUTE a statement. If this fails, it means we are in a transaction mode. Finally, we try to switch to a non-existent database. If it returns an ERROR, it means we are in session mode. If it returns a FATAL, it means we are not connected to PgBouncer at all.

## Mnemonic for this common set of psql options: "axe cutie"
postgres@scw-56578065:~/pgbouncer$ psql -Ax -qt -h /tmp test1
postgres@test1:5432=#[local:/tmp] BEGIN; ROLLBACK;
postgres@test1:5432=#[local:/tmp] PREPARE abc(int) AS SELECT $1::text;
postgres@test1:5432=#[local:/tmp] EXECUTE abc(123);
text|123
postgres@test1:5432=#[local:/tmp] \c rowdiegorow
ERROR:  No such database: rowdiegorow
## test1 is thus running in session pool_mode

postgres@scw-56578065:~/pgbouncer$ psql -Ax -qt -h /tmp test2
postgres@test2:5432=#[local:/tmp] BEGIN; ROLLBACK;
postgres@test2:5432=#[local:/tmp] PREPARE abc(int) AS SELECT $1::text;
postgres@test2:5432=#[local:/tmp] EXECUTE abc(123);
ERROR:  prepared statement "abc" does not exist
## test2 is thus running in transaction pool_mode

postgres@scw-56578065:~/pgbouncer$ psql -Ax -qt -h /tmp test3
postgres@test3:5432=#[local:/tmp] BEGIN; ROLLBACK;
ERROR:  Long transactions not allowed
## test3 is thus running in statement pool_mode

So the database-level pool modes are working as expected. PgBouncer now supports user-level pool modes as well, and these always trump the database-level pool modes. Recall that our setting for test4 in the pgbouncer.ini file was:

test4 = dbname=test3 host=/var/run/postgresql pool_mode=statement auth_user=postgres

The addition of the auth_user parameter allows us to specify other users to connect as, without having to worry about adding them to the PgBouncer auth_file. We added four sample regular users above: Alice, Bob, Eve, and Mallory. We should only be able to connect with them via auth_user, so only test4 should work:

postgres@scw-56578065:~/pgbouncer$ psql -h /tmp test1 -U alice
psql: ERROR:  No such user: alice
postgres@scw-56578065:~/pgbouncer$ psql -h /tmp test2 -U alice
psql: ERROR:  No such user: alice
postgres@scw-56578065:~/pgbouncer$ psql -h /tmp test3 -U alice
psql: ERROR:  No such user: alice
postgres@scw-56578065:~/pgbouncer$ psql -h /tmp test4 -U alice
psql (9.4.3)
Type "help" for help.

alice@test4:5432=>[local:/tmp] begin;
ERROR:  Long transactions not allowed

Let’s see if we can change the pool_mode for Alice to transaction, even if we are connected to test4 (which is set to statement mode). All it takes is a quick entry to the pgbouncer.ini file, in a section we must create called [users]:

echo "[users]
alice = pool_mode=transaction
$(cat /var/lib/postgresql/pgbouncer/etc/pgbouncer.ini)" \
> /var/lib/postgresql/pgbouncer/etc/pgbouncer.ini

## Attempt to reload pgbouncer:
postgres@scw-56578065:~/pgbouncer$ kill -HUP `head -1 /run/pgbouncer/pgbouncer.pid`

## Failed, due to a PgBouncer bug:
2001-03-05 02:27:44.376 5555 FATAL @src/objects.c:299 in function
  put_in_order(): put_in_order: found existing elem

## Restart it:
postgres@scw-56578065:~/pgbouncer$ pgbouncer /var/lib/postgresql/pgbouncer/etc/pgbouncer.ini -d

postgres@scw-56578065:~/pgbouncer$ psql -Ax -qt -h /tmp test4 -U alice
alice@test4:5432=>[local:/tmp] BEGIN; ROLLBACK;
alice@test4:5432=>[local:/tmp] PREPARE abc(int) AS SELECT $1::text;
alice@test4:5432=>[local:/tmp] EXECUTE abc(123);
ERROR:  prepared statement "abc" does not exist
## test4 is thus running in transaction pool_mode due to the [users] setting

There you have it—database-specific and user-specific PgBouncer pool_modes. Note that you cannot yet do user and database specific pool_modes, such as if you want Alice to use transaction mode for database test4 and statement mode for test5.

Connected to PgBouncer or Postgres?

2015-05-18T00:00:00+00:00

Image by David Wright

Determining if your current database connection is using PgBouncer, or going directly to Postgres itself, can be challenging, as PgBouncer is a very low-level, transparent interface. It is possible, and here are some detection methods you can use.

This was inspired by someone asking on the Perl DBD IRC channel if it was possible to easily tell if your current database handle (usually “$dbh”) is connected to PgBouncer or not. Since I’ve seen this question asked in other venues, I decided to take a crack at it.

There are actually two questions to be answered: (1) are we connected to PgBouncer, and if so, (2) what pool_mode is being run? The quickest and easiest way I found to answer the first question is to try and connect to a non-existent database. Normally, this is a FATAL message, as seen here:

$ psql testdb -p 5432
testdb=# \c ghostdb
FATAL:  database "ghostdb" does not exist
Previous connection kept
testdb=#

However, a slightly different ERROR message is returned if the same thing is attempted while connected to PgBouncer:

$ psql testdb -p 6432
testdb=# \c ghostdb
ERROR:  No such database: ghostdb
Previous connection kept
testdb=#

Thus, an ERROR will always indicate that you are connected to PgBouncer and not directly to Postgres, which will always issue a FATAL.

In the future, there will be an even simpler method. As of this writing, pgBouncer 1.6 has not been released, but it will have the ability to customize the application_name. This is a configurable session-level variable that is fairly new in Postgres. Andrew Dunstan wrote a patch which enables adding this to your pgbouncer.ini file:

application_name_add_host = 1

This will make PgBouncer modify the application_name to append some information to it such as the remote host, the remote port, and the local port. This is a feature many PgBouncer users will appreciate, as it offers an escape from the black hole of connection information that PgBouncer suffers from. Here is what it looks like on both a normal Postgres connection, and a PgBouncer connection. As you can see, this is an easier check than the “invalid database connection” check above:

## Postgres:
$ psql testdb -p 5432 -c 'show application_name'
 application_name
------------------
 psql

## PgBouncer:
$ psql testdb -p 6432 -c 'show application_name'
        application_name
--------------------------------
 psql - unix(7882@gtsm.com):6432

## DBD::Pg connections to PgBouncer get a very similar change:
$ perl testme.tmp.pl --port 6432
app - unix(6772@gtsm.com):6432

Now we have answered question of “are we connected to PgBouncer or not?”. The next question is which pool mode we are in. There are three pool modes you can set for PgBouncer, which controls when your particular connection is returned to “the pool”. For “session” mode, you keep the same Postgres backend the entire time you are connected. For “transaction”, you keep the same Postgres backend until the end of a transaction. For “statement”, you may get a new Postgres backend after each statement.

First, we can check if we are connected to PgBouncer in a statement level pool mode by taking advantage of the fact that multi-statement transactions are prohibited. PgBouncer enforces this by intercepting any attempts to enter a transaction (e.g. by issuing a BEGIN command). A very PgBouncer specific error about “Long transactions not allowed” is issued back to the client like so:

$ psql testdb -p 6432
testdb=# begin;
ERROR:  Long transactions not allowed

So, that takes care of detecting a pool_mode set to “statement”. The other two modes, transaction and session, will not give the same error. Thus, seeing that error indicates you are using a statement-level PgBouncer connection.

The next pool mode is “transaction”, which means that the server connection if released back to the pool at the end of a transaction. To figure out if we are in this mode, we take advantage of the fact that PgBouncer can be set to clean up the connection at the end of each transaction by issuing a specific command. By default, the command set by server_reset_query is DISCARD ALL, which invalidates any prepared statements, temporary tables, and other transaction-spanning, session-level items. Thus, our test will see if these session-level artifacts get discarded or not:

## Direct Postgres:
$ psql testdb -p 5432
testdb=# prepare abc(int) as select $1::text;
PREPARE
testdb=# execute abc(1);
 text
------
 1

## PgBouncer:
$ psql testdb -p 6432
testdb=# prepare abc(int) as select $1::text;
PREPARE
testdb=# execute abc(1);
ERROR:  prepared statement "abc" does not exist

Keep in mind that there are no true “transactionless” commands in Postgres. Even though we did not use a BEGIN in the psql prompt above, each command is treated as its own mini-transaction. In the case of the PgBouncer connection, the prepare is immediately followed with a DISCARD ALL, which means that our prepared statement no longer exists. Hence, we have determined that we are using a transaction-level PgBouncer connection.

Unfortunately, not getting an error does not necessarily mean your PgBouncer is NOT in transaction mode! A very negative sentence! It could be that server_reset_query is empty, meaning that temporary artifacts are not discarded at the end of the transaction. In such a case, we can take advantage of the fact that PgBouncer will allow other clients to share in our current connection, and thus be able to see the temporary items. If we create a temporary table in one pgbouncer connection, then connect again as a new client, the temporary table will only show up if we are sharing sessions but not transactions. Easier shown than explained, I suspect:

## Regular Postgres gets a fresh session:
$ psql test1 -p 5432
test1=# create temp table abc(a int);
CREATE TABLE
test1=# select * from abc;
(No rows)
test1=# ^Z ## (we suspend with CTRL-Z)
[2]+  Stopped                 psql test1 -p 5432

$ psql test1 -p 5432
test1=# select * from abc;
ERROR:  relation "abc" does not exist

## PgBouncer will re-use the same session:
$ psql test1 -p 6432
test1=# create temp table abc(a int);
CREATE TABLE
test1=# select * from abc;
(No rows)
test1=# ^Z
[2]+  Stopped                 psql test1 -p 6432

$ psql test1 -p 6432
test1=# select * from abc;
(No rows)

The final PgBouncer pool mode is “session”, and basically means the only advantage over a normal Postgres connection is the overhead to start up and connect to a new Postgres backend. Thus, the PgBouncer connections are only returned to the pool upon disconnection. The only way to tell if you are in this mode is by determining that you are not in the other two modes. :)

So, although PgBouncer is extremely transparent, there are some tricks to determine if you are connected to it, and at what pool_mode. If you can think of other (SQL-level!) ways to check, please let me know in the comments section.

Full Page Caching in Interchange 5

2013-10-28T00:00:00+00:00

I recently attended the eCommerce Innovation Conference 2013 with fellow End Pointer Richard Templet and presented on the work End Point has done to develop full-page caching in Interchange 5. The work is in final review currently for inclusion into core Interchange and should provide a roadmap for cache management in Nitesi/Interchange 6.

Parameters of a Caching Solution

In order to identify the full scope of what one means when one says an application uses caching, there are (at least) 3 aspects to define:

Where in the application stack is the cache being applied.
How long until the cache is expired.
What level of user state must it support.

The issue of cache duration is not addressed here as any such formulation will be specific to business requirements and tolerance of stale data. I will state, however, that even very brief cache durations can have an enormous impact on performance, particularly for sites that have a relatively small number of resources that absorb the bulk of the traffic. Cache durations of several minutes to several hours can drop the traffic that makes it to Interchange to a small fraction of the overall requests.

Caching and the Application Stack

Let’s examine an ideal target architecture for a simple implementation of an Interchange store, or stores. Note that we could also introduce load balancing layers to substantially boost capacity through horizontal scaling at multiple points in this stack, and if we did so we’d need to add those to our list to identify the impact of selecting a target cache point.

Browser
Reverse proxy (e.g., Varnish, Pound, nginx)
Web server (e.g., Apache httpd, nginx)
Interchange, with session storage
- Database (e.g., MySQL)
- Other applications (e.g., email server)
- Web services (e.g., payment gateway)

The higher up the stack we can cache, the more scalable our solution becomes. Interchange comes with [timed-build] which can be used to great effect to cache database results in particular, but also potentially other applications that could produce bottlenecks in performance. Moreover, because Interchange assembles the document, this is the last point in the stack that we can (directly, if at all) partially cache a resource. However, having all requests reach Interchange is, itself, a scalability issue that would have to be resolved with either horizontal scaling or pushing our cache farther up the stack.

It’s also possible to produce a static build of assets in the web server’s doc space, keeping requests from reaching Interchange at all. And while responses from a web server will have considerably less overhead and better response times than Interchange, both building and maintaining a static repository of Interchange assets is going to take some effort and, ultimately, will require horizontal scaling to relieve overload.

Our target point for the cache described herein is at the reverse proxy. We want to control our cache using standard cache headers and an nginx configuration that uses the full URL for its cache keys. The reverse proxy is chosen because:

Very fast and efficient at delivering documents.
Low maintenance as we are able to have proxy engine keep its data storage fresh according to document headers.
Assets can now be massively scaled through 3rd-party CDNs requiring no infrastructure investment and maintenance that horizontal scaling solutions lower in the stack would.
Full URL cache keys and cache headers allow us extend our cache all the way to the browser, for those user agents that respect the cache headers.

Core Interchange Support for Full-Page Caching

To provide general support for full-page caching in Interchange, we found it necessary to introduce some core features into Interchange 5. These features are in final review internally and will be committed to core soon.

SuppressCachedCookies

New boolean catalog configuration parameter that, when true, instructs Interchange not to write any cookies in a cacheable response. Cookies are inherently specific to the individual client being served and nginx will refuse to cache a resource containing Set-Cookie headers.

$::Instance->{Volatile}

Value indicates to critical core code what the request’s cache potential is. Three values:

undef - unknown, could be cached, but hasn’t been explicitly identified
true - cannot be cached. Indicates the requested resource is user-dependent and may produce different results for the same URL for different users.
false (other than undef) - explicitly treat as a “can be cached” resource. This setting can be used to reverse override other cache overrides.

The ternary nature of Volatile allows a developer to explicitly control the caching behavior of any given resource if circumstances require an adjustment to the default behavior.

[if-not-volatile]

New container tag whose body will only interpolate if the value of $::Instance->{Volatile} at the time of interpolation is false. Tag is particularly useful for placing settings for cache headers on shared resources (includes files, components, etc.) where the final document may or may not be cacheable.

OutputCookieHook

Catalog configuration parameter that takes the name of a catalog or global sub to execute just prior to Interchange writing its Set-Cookie headers. Setting was inspired by the need to maintain portions of the session on the client via cookies to allow some more stubborn session-dependent resources to be cacheable.

Obstacles to Full-Page Caching in Interchange

Standard coding practices in a typical Interchange catalog interfere with full-page caching in a number of ways, primarily:

Liberal coupling of resources with user sessions
Searches common to all site users (e.g., category lists) generate saved search objects that force session coupling
Heavy reliance on non-RESTful URLs, primarily those generated by the process actionmap

In most circumstances, these practices can be altered to produce fully cacheable resources, and particularly if the most heavily used components of the site are addressed first, such as the home page and category lists.

Catalog Changes to Mitigate Caching Obstacles

Precisely what changes are required depends on the specific coding practices used for the resources in question. However, there are a number of typical usage patterns that will to some degree affect almost all Interchange catalogs.

Use RESTful URLs

Avoid unnecessary dependence on the process actionmap, which is often used liberally precisely because it gives lots of hooks into cool and useful features. Avoid any other use of common URLs that produce varying resources based on session dependence.

Take advantage of writing custom actionmaps, which allow the developer extreme flexibility in URL construction. Because actionmaps make it easy and straightforward to produce unique URLs, they are ideal both for creating cacheable resources and fine-tuning SEO.

Permanent More for Category Lists

By default, search objects which Interchange uses for more lists, are restricted to access from the generating user’s session. This is a safeguard as often search results include personal data for access only to the requestor. However, for features such as category lists, this creates a difficult burden for the developer who wishes to cache the popular resources and whose results are identical across all users.

We can overcome this difficulty by making the search definitions for category lists, or other canned searches, include the permanent more indicator. Permanent more causes all identical searches to share a common search object accessible by the same URLs, and freeing the usual coupling with the session of the search originator.

Address Common Session Variables

There are certain session variables that are often found in page code and can cause a number of difficulties when trying to make a resource cacheable. Start by tracking through the use of each of the following and come up with a strategy to remove the dependencies so that the interpolated code is free of them:

[value]
[data session ___]
[set]/[seti]
Any [if] conditionals that use those respective bases (e.g., [if value], [if session], and [if scratch])

Cacheable Redirects

Any code that issues a redirect must do so consistently with respect to its URL. Any redirect will cache the http code and, if issued conditionally, will force all users accessing the cached resource to also redirect. This practice is seen often in Interchange catalogs, particularly when monitoring pages that are restricted to logged-in users. In summary, it’s OK to cache redirects, but just make sure that a given URL is either always, or never, redirected.

Profile and “Click” Code

It is common practice to define profile and click code in scratch variables. This is particularly true for click code defined with the [button] tag, which while convenient causes the click action to be defined under the hood in scratch space. In order for these event-driven features to work, the resource must compile that code and seed it in the session in anticipation of the user’s next actions. If those resources are cached, those important features are never added to the session as the result of a page load and, so, none of the actions will work.

All use of [button] or [set] to produce click or profile code should be moved into the profile files (typically found in etc/profile.*). There they are added to the Interchange global configuration at compile time and are thus available to all users without regard to the state of their sessions. This is good practice generally since it is often easy (particularly with [button]) to have multiple actions map to the same scratch key. When that happens, a user going through the browser back button can get invalid results on actions taken because the click or profile definitions have changed with respect to the anticipated such actions on the current page.

Set SuppressCachedCookies to Yes

As described earlier, this will tell the Interchange core not to write any cookies if the resource is to be cached.

Define Cache-Control Headers

At any point in the development of the response body, Interchange can be issued a pragma that tells it to treat the response as cacheable. This will interact with the core features described above to ensure that there is no impact on the user session as a result of this request, and put the correct headers in the output stream (as well as keep the cookie headers out).

Invocation looks something like [tag pragma cache_control]max-age=NNN[/tag], where NNN is the number of seconds the cache should persist.

Impact on Session Management

Any resource considered cacheable should a priori have neither impact nor dependence on a session. This must be true if we consider that, once cached, a user will interact with the page—and expect correct behavior—without ever touching Interchange. This introduces some new conditions associated with the session:

The initial user request against a cacheable resource will not generate a session. Why is this so? Apart from the already-noted discrepancy of accessing the resource for a cached vs. live hit, generating a session would necessitate producing the session cookie. Returning that cookie would invalidate the resource as cacheable. Further, one of the significant advantages of a reverse-proxy caching strategy is to provide protection in a DoS attack, and the user agents in such an attack are very unlikely to maintain a session. Thus, if we were failing to produce a cache on initial hits to allow setting a session, all those DoS hits would reach Interchange, and on top of that be churning out session entries on the server.
Session writing is suppressed on any response with a cacheable resource. Interchange must treat the response without any permanence because all accesses of the resource from the cache will never reach Interchange. If the request that produced the cache also wrote that user’s session, it would produce a deviation in behavior between the cached v. live requests.

Overrides on a Cacheable Response

Any action resulting in a POST is considered to imply the necessity of the user initiating the request reaching the session (or the database, or some other permanent storage controlled by the Interchange app). Thus POSTs by default force the Volatile setting to true. However, note this can be overridden by the developer if necessary (e.g., if a DoS hits the home page with a POST rather than the expected GET).

Similarly, any requests passing through either the “process” or “order” actionmap are assumed to require access to the session. “process” will most often be issued as a POST as well, although using “order” with a GET is common.

User State on Cacheable Resources

A big mistake a developer may make when considering full-page caching is to assume an all-or-nothing approach. Trying to compartmentalize an entire catalog into fully cacheable resources would be a daunting task, requiring essentially the construction of a fully client-side application and session management. This is neither realistic nor desirable.

A catalog can gain considerable benefit simply from evaluating those resources which do not require session entanglement at all and starting with them. Without considering users that are logged in or have items in their cart, under most circumstances the home and category list pages should be free from entanglement. With a bit of URL management, the resources can skip the cache when a user is logged in or has items in the cart.

“Read Only” User State

If caching is desirable on resources that cannot be decoupled from session influence, we can expose the necessary parts of the session to the client in the form of cookies and can refactor our document to contain client-side code to manage the session use. Typical examples of this would be personalization for logged in users, or the display of a small cart on all pages. The session data stored in the cookie is controlled exclusively by Interchange and is read-only on the client. Each time the session is accessed and updated, the cookie is re-written to the client.

Management of such a process is relatively easy with modern JavaScript frameworks such as jQuery. As a typical example, one might need to replace the following session-dependent code

[if session logged_in]
  Hi, [value fname]!
[/if]

with client-side management:

<span id="fname_display"></span>

and in the ready() event elsewhere with our session cookie data stored in valuesData:

if (valuesData.fname) {
  $('#fname_display').replaceWith('Hi, ' + valuesData.fname + '!');
}

The OutputCookieHook was developed as a convenient mechanism for constructing the proposed valuesData cookie above, allowing for a subroutine to build the cookie just prior to the core code that constructs the document headers but after any standard actions that would alter the session and would need to be captured in the cookie data.

“Read Write” User State

If state needs are more complex on a particularly popular resource, it may be necessary to allow our state cookie to also be updated from the client. With the tools described here, the developer can either amend the existing cookie, or construct a new one, that captures data input by the client through subsequent requests to cached resources. Once the user issues the next non-cached request, an Autoload subroutine could be constructed to identify that changes have occurred and then sync those changes back to the user’s session. While implementing read-write user state may be challenging, it is possible and has been done at End Point for clients where that need exists.

Hopefully this provides a good idea of how to get started when approaching full-page caching, which is possible in most web app frameworks, and soon will be much easier in Interchange 5 with the new core tools introduced here.

Interchange Search Caching with “Permanent More”

2012-01-02T00:00:00+00:00

Most sites that use Interchange take advantage of Interchange’s “more lists”. These are built-in tools that support an Interchange “search” (either the search/scan action, or result of direct SQL via [query]) to make it very easy to paginate results. Under the hood, the more list is a drill-in to a cached “search object”, so each page brings back a slice from the cache of the original search. There are extensive ways to modify the look and behavior of more lists and, with a bit of effort, they can be configured to meet design requirements.

Where more lists tend to fall short, however, is with respect to SEO. There are two primary SEO deficiencies that get business stakeholders’ attention:

There is little control over the construction of the URLs for more lists. They leverage the scan actionmap and contain a hash key for the search object and numeric data to identify the slice and page location. They possess no intrinsic value in identifying the content they reference.
The search cache by default is ephemeral and session-specific. This means all those results beyond page 1 the search engine has cataloged will result in dead links for search users who try to land directly on the more-listed pages.

It is the latter issue that I wish to address because there is—and has been for some time now—a simple mechanism called “permanent more” to remedy the default behavior.

You can leverage “permanent more” by adding the boolean mv_more_permanent, or the shorthand pm, to your search conditions. E.g.:

Link:

    <a href="[area search="
        co=1
        sf=category
        se=Foo
        op=rm
        more=1
        ml=5
        <b>pm=1</b>
    "]">All Foos</a>

Loop:

    [loop search="
        co=1
        sf=category
        se=Foo
        op=rm
        more=1
        ml=5
        <b>pm=1</b>
    "]
    ...loop body with [more-list]...
    [/loop]

Query:

    [query
        list=1
        more=1
        ml=10
        <b>pm=1</b>
        sql="SELECT * FROM products WHERE category LIKE '%Foo%'"
    ]
    ...same as loop but with 10 matches/page...
    [/query]

If the initial search is defined with the “permanent more” setting, it will produce the following adjustments:

The hash key used to store and identify the search cache is deterministic based on the search conditions. Many searches for Interchange are category driven. Thus, all end users who wish to browse a category end up clicking identical links, which create duplicate search caches, belonging uniquely to them. With permanent more, they all share the same cache, with the same identifier. As long as the search conditions don’t change, neither does the cache identifier. Even as the cache is refreshed with new executions of the search, the object remains in the same location. Thus, the results a search engine produced this morning reference links still valid now, tomorrow, or next week, provided they reference the same search conditions.
The cached search object has no session affinity. Any link referencing the cache with the correct hash key has access to the content.

Taken together, “permanent more” removes (for the most part, addressed later) dead links from more lists cataloged by search engines. There are, however, other benefits to “permanent more” beyond those intended as described above:

As stated in passing, standard Interchange search caching produces duplicate search objects for common search conditions. For a busy site, these caches can have an impact on storage. Typically, maintenance is implemented to clean up cache files for all such files whose age exceeds by some amount the session duration (standard is 48 hours). With permanent more, duplicate caches are eliminated. A cache location is reused by all users with the same search requirements, keeping data-storage requirements for caches to the minimum necessary. As searches change, ophaned caches can still easily be cleaned up as they will immediately start to age with no more access to them necessary for storage.
For the same reason that “permanent more” resolves search-engine links, it also resolves content management for individual sites using a reverse proxy for caching. Because most (and certainly the easiest) caching keys are based off of URL, the deterministic nature of the hash keys for “permanent more” allows assurance that the cached content in the proxy accurately reflects the search content over time, and that all users will hit the cached resource and not generate new, unique links with varying hash keys.

One shortcoming of “permanent more” to be aware of is the impact of changing data underneath the search. Even if search conditions do not change, the count and order of matching record sets may. So, e.g., enough products may be removed from a given category to cause the last page of a more list to become empty, which would cause any specific link into that page to become dead. More minor, but still a possibility, is the introduction or removal of products so that a particularly searched-for term has been “bumped” to another page within the search cache since the last time the search engine crawled the more lists. For searches backed by particularly volatile data, “permanent more” may not be sufficient to address search-engine or caching demands.

Finally, “permanent more” should be avoided for any search features that may cache data sensitive to an individual user. This is unlikely to happen as, under most circumstances, the configuration of the search itself will change based on the unique characteristics of the user executing the search (e.g., a username included in a query to review order history). However, it is still possible that context-sensitive information could be stored in the search object and, if so, all other users with access to the more lists would have access to that information.

Spree Performance Benchmarking

2011-05-25T00:00:00+00:00

I see a lot of questions regarding Spree performance in the spree-user group, but they are rarely answered with metrics. I put together a quick script using the generic benchmark tool ab to review some data. Obviously, the answer to how well a site performs and scales is highly dependent on the host and the consumption of the web application, so the data here needs to be taken with a grain of salt. Another thing to note is that only two of the following use cases are running on Rails 3.0 — many of our current Spree clients are on Spree 0.11.2 or older. I also included one non-Spree Rails ecommerce application, in addition to a few non-Rails applications for comparison. All of the tests were run from my home network, so in theory there shouldn’t be bias on performance tests for sites running on End Point servers.

	ab -n 100
	-c 2 homepage	-c 20 homepage	-c 2 product page	-c 20 product page
Client #1 Spree: 0.11.2 Hosting: 4 cores, 512 GB RAM DB: MySQL # Products: <100	7.49	24.75	6.49	19.87	Requests per second
266.889	808.041	307.997	1006.552	Time per request (ms)
Client #2 Spree 0.11.2 Hosting: Engineyard, medium instance DB: MySQL # Products: 100s	5.32	20.28	5.36	18.03	Requests per second
375.713	986.309	373.289	1109.524	Time per request (ms)
Client #3 Spree: 0.9.0 Hosting: 4 cores, 1 GB RAM DB: PostgreSQL # Products: <100	4.91	25.39	1.98	6.54	Requests per second
407.135	787.782	1011.875	3060.062	Time per request (ms)
(Former) Client #4 Spree: 0.11.2 Hosting: Unknown DB: PostgreSQL # Products: >5000	20.69	8.84	10.15	19.28	Requests per second
96.673	2262.105	196.996	1037.146	Time per request (ms)
Client #5 Spree: 0.11.2 Hosting: EngineYard, small instance DB: MySQL # Products: 1	12.28	16.23	N/A	N/A	Requests per second
162.909	1231.945	N/A	N/A	Time per request (ms)
Client #6 Spree: 0.40 Hosting: 4 cores, 1 GB RAM DB: MySQL # Products: 50-100	3.61	8.93	2.96	3.06	Requests per second
553.569	2240.657	675.306	6539.433	Time per request (ms)
SpreeDemo Spree: Edge Hosting: Heroku, 2 dynos DB: Unknown # Products: 100s	8.17	12.79	4.7	5.48	Requests per second
244.831	1563.642	425.27	3652.927	Time per request (ms)
Client #7 *custom Rails ecommerce app Hosting: 1.0 GB RAM DB: MySQL # Products: 1000s	5.43	29.8	4.45	23.14	Requests per second
368.409	671.082	448.962	864.24	Time per request (ms)
Interchange Demo Hosting: 4 cores, 2 GB RAM DB: MySQL # Products: >500	7.41	55.27	7.5	13.93	Requests per second
269.942	361.875	266.492	1435.51	Time per request (ms)
Client #8 *PHP site, serves fully cached pages with nginx with no app server or db hits Hosting: 4 cores, 4 GB RAM	10.81	30.54	6.05	9.87	Requests per second
184.994	654.858	330.727	2027.092	Time per request (ms)
Magento Demo Hosting: Unknown DB: Unknown # Products: 100s	4.26	44.85	2.68	36.29	Requests per second
469.831	445.931	745.472	551.11	Time per request (ms)

Here’s the same data in graphical form:

Requests per Second

Time Per Request (ms)

We expect to see high performance on some of the sites with significant performance optimization. On smaller VPS, we expect to see the the server choke with higher concurrency.

Surge 2010 wrap-up

2010-10-05T00:00:00+00:00

Following up on my earlier post about day 1 of the conference, here is an unsorted collection of what I felt were noteworthy observations made in talks at Surge 2010:

Web engineering as a separate discipline from computer science or software development started around 1999. It is interdisciplinary, involving human factors engineering, systems engineering, operations research, fault-tolerant design, and control systems engineering. (John Allspaw)

A real-time system is one in which the correctness of a system is tied to its timeliness. Eventual consistency is an oxymoron if timeliness is part of the data itself. Caching by CDNs can’t solve our problems here. (Bryan Cantrill)

Pre-fab metrics are worth less (and maybe worthless) when not tied to something in your business. Message queues enable lots of new uses because of the ability to have multiple observers. See Esper (Java, GPL) for live ongoing SQL-like queries of messages from AMQP sources, etc. (Theo Schlossnagle)

On scaling up vs. out: If your numbers show “up” is enough, be happy you can keep your system simpler. (Theo Schlossnagle)

Anyone can only ever know the past in a distributed system. There’s no such thing as global state in reality. Our systems are always at least slightly inconsistent with the world. “Eventually consistent” just acknowledges the reality of delay and focuses on measuring and dealing with that. (Justin Sheehy)

Reliability compared to resiliency: Being resilient means success of your mission despite partial failure of components. How do you deal with failure? Degrade, and know before your users do. (Justin Sheehy)

Build in monitoring during development, so it’s not a bottleneck right before deployment. (John Allspaw)

Data comes from the devil. Models come from God. Data + Models = Insight. Data needs to be put in a prison (a model) and made to confess the truth. Measurement is a process. Numbers aren’t right. What is the error range? Visualization is helpful, but analyze the raw data looking for anomalies (such as > 100% efficiency, etc.). VAMOOS = visualize, analyze, modelize, over & over till satisfied. (Neil Gunther)

Anycast for DNS alone tends to localize on the user’s recursive resolver, not their actual location. Anycast for the actual content delivery automatically localizes on the user’s actual location. (Tom Daly)

To scale up, add more capacity to do X, make system do X faster, or stop doing so much X. What makes a task take time? It’s utilizing a resource, it’s waiting for a response, or it’s waiting for synchronization. “Shard early & often” is expensive & unnecessary for most situations. Sharding makes sense when write demand exceeds capacity. (Baron Schwartz)

Eight fallacies of distributed computing were discussed by Ruslan Belkin.

Mike Mallone of SimpleGeo gave a fascinating talk on working with geolocation data in Cassandra. It’d be wonderful to see an open source release of their order-preserving partitioner that allows for range queries in a single dimension. Or to start with, just the slides from Mike’s talk!

In summary, it was a very good conference!

Surge 2010 day 1

2010-10-01T00:00:00+00:00

Today (technically, yesterday) was the first day of the Surge 2010 conference in Baltimore, Maryland. The Tremont Grand venue is perfect for a conference. The old Masonic lodge makes for great meeting rooms, and having a hallway connect it to the hotel was nice to avoid the heavy rain today. The conference organization and scheduling and Internet have all been solid. Well done!

There were a lot of great talks, but I wanted to focus on just one that was very interesting to me: Artur Bergman’s on scaling Wikia. Some points of interest:

They (ab)use Google Analytics to track other things besides the typical pages viewed by whom, when. For example, page load time as measured by JavaScript, with data sent to a separate GA profile for analysis separately from normal traffic. That is then correlated with exit rates to give an idea of the benefit of page delivery speed in terms of user stickiness.
They use the excellent Varnish reverse proxy cache.
500 errors from the origin result in a static page served by Varnish, with error data hitting a separate Google Analytics profile.
They have both geographically distributed servers and team.
They’ve found SSDs (solid state disks) to be well worth the extra cost: fast, using less power in a given server, and requiring fewer servers overall. They have to use Linux software RAID because no hardware RAID controllers they’ve tested could keep up with the speed of SSD. They have run into the known problems with disk write performance dropping as they fill and recycle, but haven’t found it to be a problem when used on replaceable cache machines.
They run their own CDN, with nodes running Varnish, Quagga (for BGP routing), BIND, and Scribe. But they use Akamai for static content.
Even running Varnish with 1 second TTL can save your backend app servers when heavy traffic arrives! One hit per second is no problem; thousands may mean meltdown.
Serving stale cached content when the backend is down can be a good choice. It means most visitors will never know anything was wrong. (Depends on the site’s functions, of course.)
Their backup datacenter in Iowa is in a former nuclear bunker. See monitoring graphs for it.
Wikia ops staff interact with their users via IRC. This “crowdsourced monitoring” has resulted in a competition between Wikia ops people and the users to see who can spot outages first.
Having their own hardware in multiple redundant datacenters has meant much more leverage in pricing discussions with datacenters. “We can just move.”
They own their own hardware, and run on bare metal. At no time does user traffic pass through any virtualized systems at all. The performance just isn’t there. They do use virtual machines for some external monitoring stuff.
They use Riak for N-master inter-datacenter synchronization, and RiakFS for sessions and files. RiakFS is for the “legacy” MediaWiki need for POSIX access to files, but they can serve those files to the general public from Riak’s HTTP interface via Varnish cache.
They use VPN tunnels between datacenters. Sometimes using their own routes, even over multiple hops, leads to faster transit than going over the public Internet.
Lots of interesting custom VCL (Varnish Configuration Language) examples.

This had plenty of interesting things to consider for any web application architecture.

Distributed Transactions and Two-Phase Commit

2010-07-29T00:00:00+00:00

The typical example of a transaction involves Alice and Bob, and their bank. Alice pays Bob $100, and the bank needs to debit Alice and credit Bob. Easy enough, provided the server doesn’t crash. But what happens if the bank debits Alice, and then before crediting Bob, the server goes down? Or what if they credit Bob first, and then try to debit Alice only to find she doesn’t have enough funds? A transaction allows the debit and credit operations to happen as a package (“atomically” is the word commonly used), so either both operations happen or neither happens, even if the server crashes halfway through the transaction. That way the bank never credits Bob without debiting Alice, or vice versa.

That’s simple enough, but the situation can become more complex. What if, for instance, for buzzword-compliance purposes, the bank has “sharded” its accounts database by splitting it in pieces and putting each piece on a different server (whether this is would be smart or not is outside the scope of this post). The typical transaction handles statements issued only for one database, so we can’t wrap the debit and credit operations within a single BEGIN/COMMIT if Alice’s account information lives on one server and Bob’s lives on another.

Enter “distributed transactions”. A distributed transaction allows applications to group multiple transaction-aware systems into a single transaction. These systems might be different databases, or they might include other systems such as message queues, in which case the transaction concept means a message would get delivered if and only if the rest of the transaction completed. So with a distributed transaction, the bank could debit Alice’s account in one database and credit Bob’s in another, atomically.

All this comes at some cost. Distributed transactions require a “transaction manager”, an application which handles the special semantics required to commit a distributed transaction. Second, the systems involved must support “two-phase commit” (which was added to PostgreSQL in version 8.1). Distributed transactions are committed using PREPARE TRANSACTION ‘foo’ (phase 1), and COMMIT PREPARED ‘foo’ or ROLLBACK PREPARED ‘foo’ (phase 2), rather than the usual COMMIT or ROLLBACK.

The beginning of a distributed transaction looks just like any other transaction: the application issues a BEGIN statement (optional in PostgreSQL), followed by normal SQL statements. When the transaction manager is instructed to commit, it runs the first commit phase by saying “PREPARE TRANSACTION ‘foo’” (where “foo” is some arbitrary identifier for this transaction) on each system involved in the distributed transaction. Each system does whatever it needs to do to determine whether or not this transaction can be committed and to make sure it can be committed even if the server crashes, and reports success or failure. If all systems succeed, the transaction manager follows up with “COMMIT PREPARED ‘foo’”, and if a system reports failure, the transaction manager can roll back all the other systems using either ROLLBACK (for those transactions it hasn’t yet prepared), or “ROLLBACK PREPARED ‘foo’”. Using two-phase commit is obviously slower than committing transactions on only one database, but sometimes the data integrity it provides justifies the extra cost.

In PostgreSQL, two-phase commit is supported provided max_prepared_transactions is nonzero. A PREPARE TRANSACTION statement persists the current transaction to disk, and dissociates it from the current session. That way it can survive even if the database goes down. The current session no longer has an active transaction. However, the prepared transaction acts like any other open transaction in that all locks held by the prepared transaction remain held, and VACUUM cannot reclaim storage from that transaction. So it’s not a good idea to leave prepared transactions open for a long time.

Distributed transactions are most common, it seems, in Java applications. Full J2EE application servers typically come with a transaction manager component. For my examples I’ll use an open source, standalone transaction manager, called Bitronix. I’m not particularly fond of using Java for simple scripts, though, so I’ve used JRuby for this demonstration code.

This script uses two databases, which I’ve called “athos” and “porthos”. Each has same schema, which provides a simple framework for the sharded bank example described above. This schema provides a table for account names, another for ledger information, and a simple trigger to raise an exception when a transaction would bring a person’s balance below $0. I’ll first populate athos with Alice’s account information. She gets $200 to start. Bob will go in the porthos database, with no initial balance.

5432 josh@athos# insert into accounts values ('Alice');
INSERT 0 1
5432 josh@athos*# insert into ledger values ('Alice', 200);
INSERT 0 1
5432 josh@athos*# commit;
COMMIT5432 josh@athos# \c porthos
You are now connected to database "porthos".
5432 josh@porthos# insert into accounts values ('Bob');
INSERT 0 1
5432 josh@porthos*# commit;
COMMIT

Use of Bitronix is pretty straightforward. After setting up a few constants for easier typing, I create a Bitronix data source for each PostgreSQL database. Here I have to use the PostgreSQL JDBC driver’s org.postgresql.xa.PGXADataSource class; “XA” is Java’s protocol for two-phase commit, and requires JDBC driver support. Here’s the code for setting up one data source; the other is just the same.

ds1 = PDS.new
ds1.set_class_name 'org.postgresql.xa.PGXADataSource'
ds1.set_unique_name 'pgsql1'
ds1.set_max_pool_size 3
ds1.get_driver_properties.set_property 'databaseName', 'athos'
ds1.get_driver_properties.set_property 'user', 'josh'
ds1.init

Then I simply get a connection from each data source, instantiate a Bitronix TransactionManager object, and begin a transaction.

c1 = ds1.get_connection
c2 = ds2.get_connection
btm = TxnSvc.get_transaction_manager
btm.begin

Within my transaction, I just use normal JDBC commands to debit Alice and credit Bob, after which I commit the transaction through the TransactionManager object. If this transaction fails, it raises an exception, which I can capture using Ruby’s begin/rescue exception handling, and roll back the transaction.

begin
  s2 = c2.prepare_statement "INSERT INTO ledger VALUES ('Bob', 100)"
  s2.execute_update
  s2.close

  s1 = c1.prepare_statement "INSERT INTO ledger VALUES ('Alice', -100)"
  s1.execute_update
  s1.close

  btm.commit
  puts "Successfully committed"
rescue
  puts "Something bad happened: " + $!
  btm.rollback
end

When I run this, Bitronix gives me a bunch of output, which I haven’t bothered to suppress, but among it all is the “Successfully committed” string I told it to print on success. Since Alice is debited $100 each time we run this, and she started with $200, we can run it twice before hitting errors. On the third time, we get this:

Something bad happened: org.postgresql.util.PSQLException: ERROR: Rejecting operation; account owner Alice’s balance would drop below 0

This is our trigger firing, to tell us that we can’t debit Alice any more. If I look in the two databases, I can see that everything worked as planned:

5432 josh@athos*# select get_balance('Alice');
 get_balance 
-------------
           0
(1 row)

5432 josh@athos*# \c porthos 
You are now connected to database "porthos".
5432 josh@porthos# select get_balance('Bob');
 get_balance 
-------------
         200
(1 row)

Remember I’ve run my script three times, but Bob has only been credited $200, because that’s all Alice had to start with.

PostgreSQL: Migration Support Checklist

2010-07-22T00:00:00+00:00

A database migration (be it from some other database to PostgreSQL, or even from an older version of PostgreSQL to a nice shiny new one) can be a complicated procedure with many details and many moving parts. I’ve found it helpful to construct a list of questions in order to make sure that you’re considering all aspects of the migrations and gauge the scope of what will be involved. This list includes questions we ask our clients; feel free to contribute your own additional considerations or suggestions.

Technical questions:

Database servers: How many database servers do you have? For each, what are the basic system specifications (OS, CPU architecture, 32- vs 64-bit, RAM, disk, etc)? What kind of storage are you using for the existing database, and what do you plan to use for the new database? Direct-attached storage (SAS, SATA, etc.), SAN (what vendor?), or other? Do you use any configuration management system such as Puppet, Chef, etc.?
Application servers and other remote access: How many application servers do you have? For each, what are the basic system specifications (OS, CPU architecture, 32- vs 64-bit, RAM, disk, etc)? Do you use any configuration management system such as Puppet, Chef, etc.? What other network considerations are there? Is ODBC used, or SSL transport, any VPNs? Are multiple datacenters involved? How about egress/ingress firewalls?
Middleware: Do you currently use any sort of connection pooling, load balancing, or other middleware between your application and database servers?
Data needs: Can you describe your data access patterns? i.e., is the majority of your data historical and rarely accessed? Are there any existing reporting needs that will need to be duplicated on the PostgreSQL system? Do you already have reports of database usage, including traffic levels, frequent or intensive queries, etc?
Size: What kind of transaction volume do you see? How large are your databases? How many tables do you have and what is the size of the larger ones? How many users or database connections will you need to support?
Backups: What are your current backup policies/procedures? How will these need to change with the move to PostgreSQL?
Replication/load balancing: What kind of system redundancy do you currently have/need? Do you have any kind of database load-balancing or master-slave replication?
Monitoring: What is the current monitoring/in-house support infrastructure? What needs to be duplicated, and can any portion of this facility be reused?
Interfaces: What language are your applications written in, and what drivers exist to connect to your current database? Will there be a compatible driver available in your language of choice in order?
Extensions: Are you currently using any in-database procedures or functionality (i.e., in PL/SQL or another embedded language of choice)? If so, how many? What will the difficulty be in porting these functions to PostgreSQL?

And a couple of business-related questions:

Scheduling: What is the timeframe for transition? When can appropriate downtime be scheduled? How much database downtime can you afford?
Staffing: Do you currently have in-house DBAs to manage the servers, etc on a day-to-day basis? Is there anyone with PostgreSQL experience or familiarity on staff?

Being able to answer all of these questions is critical to formulating a migration plan and carrying out a migration successfully.

Particularly with the impending (July 2010) end of life for previous PostgreSQL releases 7.4, 8.0 and (in November 2010) 8.1, a database migration may be on your radar. End Point is one of many professional PostgreSQL support companies who would be happy to assist you in your transition.

Cassandra, Thrift, and Fibers in EventMachine

2010-05-08T00:00:00+00:00

I’ve been working with Cassandra and EventMachine lately, in an attempt to maximize write throughput for bulk loading situations (and I would prefer to not abandon the pretty Ruby classes I have fronting Cassandra, hence EventMachine rather than hopping over to Java or Scala).

The Thrift client transport for EventMachine requires the use of fibers. The documentation available for how fibers and EventMachine interact is not all that clear just yet, so perhaps documenting my adventures will be of use to somebody else.

A single fiber is traditionally imperative

EventMachine puts the I/O on background threads, but your use of the I/O interface will interact with it as if it’s a traditional blocking operation.

#!/usr/bin/env ruby

require 'eventmachine'
require 'thrift_client'
require 'thrift_client/event_machine'
require 'cassandra'

def get_client 
  Cassandra.new('Keyspace1',
                '127.0.0.1:9160',
                :transport_wrapper => nil,
                :transport         => Thrift::EventMachineTransport)
end

def write(client, key, hash)
  puts "Writing #{key}."
  client.insert('Standard1', key, hash)
  puts "Wrote #{key}."
end

EM.run do
  Fiber.new do
    client = get_client
    write(client, 'foo', {'aard' => 'vark'})
    write(client, 'bar', {'platy' => 'pus'})
    EM.stop
  end.resume
end

The Thrift::EventMachine transport performs the actual Thrift network operations (connecting, sending data, receiving data) on a fiber in one of EventMachine’s background threads. But it manages the callbacks and errbacks internally so the client behaves in usual blocking manner and does not expose the asyncronous delights going on behind the scenes.

Therefore, in the code snippet above, the “foo” row will be inserted first, and then the “bar” row. Every time. The output always is:

Wrote foo.
Wrote bar.

The above snippet is contrived, but it makes an import point: given two or more Thrift operations (like Cassandra inserts) that are logically independent of each other such that their order does not matter, you’re not necessarily gaining a lot if those operations happen in the same fiber.

For concurrency, use multiple fibers

Now let’s replace the above code sample’s EM.run block with this:

EM.run do
  @done = 0 
  Fiber.new do
    write(get_client, 'foo', {'aard' => 'vark'})
    @done += 1
  end.resume
  Fiber.new do
    write(get_client, 'bar', {'platy' => 'pus'})
    @done += 1                 
  end.resume                   
  EM.add_periodic_timer(1) { EM.stop if @done == 2 } 
end

You don’t know how this is going to play out, but the typical output proves the concurrent operation of the two fibers involved:

Writing foo.
Writing bar.
Wrote foo.
Wrote bar.

If we were writing a larger number of rows out to Cassandra, we could expect to see a greater variety of interleaving between the respective fibers.

Note a critical difference between the two examples. In the single-fiber example, we issue the EM.stop as the final step of the fiber. Because the single fiber proceeds serially, this makes sense. In the multi-fiber example, things run asyncronously, so we have no way of knowing for sure which fiber will complete first. Consequently, it’s necessary have some means of signifying that work is done and the EM can stop; in this lame example, the @done instance variable acts as this flag. In a more rigorous example, you might use a queue and a queue’s size to organize such things.

PostgreSQL EC2/EBS/RAID 0 snapshot backup

2010-02-23T00:00:00+00:00

One of our clients uses Amazon Web Services to host their production application and database servers on EC2 with EBS (Elastic Block Store) storage volumes. Their main database is PostgreSQL.

A big benefit of Amazon’s cloud services is that you can easily add and remove virtual server instances, storage space, etc. and pay as you go. One known problem with Amazon’s EBS storage is that it is much more I/O limited than, say, a nice SAN.

To partially mitigate the I/O limitations, they’re using 4 EBS volumes to back a Linux software RAID 0 block device. On top of that is the xfs filesystem. This gives roughly 4x the I/O throughput and has been effective so far.

They ship WAL files to a secondary server that serves as warm standby in case the primary server fails. That’s working fine.

They also do nightly backups using pg_dumpall on the master so that there’s a separate portable (SQL) backup not dependent on the server architecture. The problem that led to this article is that extra I/O caused by pg_dumpall pushes the system beyond its I/O limits. It adds both reads (from the PostgreSQL database) and writes (to the SQL output file).

There are several solutions we are considering so that we can keep both binary backups of the database and SQL backups, since both types are valuable. In this article I’m not discussing all the options or trying to decide which is best in this case. Instead, I want to consider just one of the tried and true methods of backing up the binary database files on another host to offload the I/O:

Create an atomic snapshot of the block devices
Spin up another virtual server
Mount the backup volume
Start Postgres and allow it to recover from the apparent “crash” the server had (since there wasn’t a clean shutdown of the database before the snapshot
Do whatever pg_dump or other backups are desired
Make throwaway copies of the snapshot for QA or other testing

The benefit of such snapshots is that you get an exact backup of the database, with whatever table bloat, indexes, statistics, etc. exactly as they are in production. That’s a big difference from a freshly created database and import from pg_dump.

The difference here is that we’re using 4 EBS volumes with RAID 0 striped across them, and there isn’t currently a way to do an atomic snapshot of all 4 volumes at the same time. So it’s no longer “atomic” and who knows what state the filesystem metadata and the file data itself would be in?

Well, why not try it anyway? Filesystem metadata doesn’t change that often, especially in the controlled environment of a Postgres data volume. Snapshotting within a relatively short timeframe would be pretty close to atomic, and probably look to the software (operating system and database) like some kind of strange crash since some EBS volumes would have slightly newer writes than others. But aren’t all crashes a little unpredictable? Why shouldn’t the software be able to deal with that? Especially if we have Postgres make a checkpoint right before we snapshot.

I wanted to know if it was crazy or not, so I tried it on a new set of services in a separate AWS account. Here are the notes and some details of what I did:

1. Created one EC2 image:

Amazon EC2 Debian 5.0 lenny AMI built by Eric Hammond
Debian AMI ID ami-4ffe1926 (x86_64)
Instance Type: High-CPU Extra Large (c1.xlarge) — 7 GB RAM, 8 CPU cores

2. Created 4 x 10 GB EBS volumes

3. Attached volumes to the image

4. Created software RAID 0 device:

mdadm -C /dev/md0 -n 4 -l 0 -z max /dev/sdf /dev/sdg /dev/sdh /dev/sdi

5. Created XFS filesystem on top of RAID 0 device:

mkfs -t xfs -L /pgdata /dev/md0

6. Set up in /etc/fstab and mounted:

mkdir /pgdata
# edit /etc/fstab, with noatime
mount /pgdata

7. Installed PostgreSQL 8.3

8. Configured postgresql.conf to be similar to primary production database server

9. Created empty new database cluster with data directory in /pgdata

10. Started Postgres and imported a play database (from public domain census name data and Project Gutenberg texts), resulting in about 820 MB in data directory

11. Ran some bulk inserts to grow database to around 5 GB

12. Rebooted EC2 instance to confirm everything came back up correctly on its own

13. Set up two concurrent data-insertion processes:

50 million row insert based on another local table (INSERT INTO … SELECT …), in a single transaction (hits disk hard, but nothing should be visible in the snapshot because the transaction won’t have committed before the snapshot is taken)
Repeated single inserts in autocommit mode (Python script writing INSERT statements using random data from /usr/share/dict/words piped into psql), to verify that new inserts made it into the snapshot, and no partial row garbage leaked through

14. Started those “beater” jobs, which mostly consumed 2-3 CPU cores

15. Manually inserted a known test row and created a known view that should appear in the snapshot

16. Started Postgres’s backup mode that allows for copying binary data files in a non-atomic manner, which also does a CHECKPOINT and thus also a filesystem sync:

SELECT pg_start_backup('raid_backup');

17. Manually inserted a 2nd known test row & 2nd known test view that I don’t want to appear in the snapshot after recovery

18. Ran snapshot script which calls ec2-create-snapshot on each of the 4 EBS volumes—during first run, run serially quite slowly taking about 1 minute total; during second run, run in parallel such that the snapshot point was within 1 second for all 4 volumes

19. Tell Postgres the backup’s over:

SELECT pg_stop_backup();

20. Ran script to create new EBS volumes derived from the 4 snapshots (which aren’t directly usable and always go into S3), using ec2-create-volume --snapshot

21. Run script to attach new EBS volumes to devices on the new EC2 instance using ec2-attach-volume

22. Then, on the new EC2 instance for doing backups:

mdadm --assemble --scan
mount /pgdata
Start Postgres
Count rows on the 2 volatile tables; confirm that the table with the in-process transaction doesn’t show any new rows, and that the table getting individual rows committed to reads correctly
VACUUM VERBOSE — and confirm no errors or inconsistencies detected
pg_dumpall # confirmed no errors and data looks sound

It worked! No errors or problems, and pretty straightforward to do.

Actually before doing all the above I first did a simpler trial run with no active database writes happening, and didn’t make any attempt for the 4 EBS snapshots to happen simultaneously. They were actually spread out over almost a minute, and it worked fine. With the confidence that the whole thing wasn’t a fool’s errand, I then put together the scripts to do lots of writes during the snapshot and made the snapshots run in parallel so they’d be close to atomic.

There are lots of caveats to note here:

This is an experiment in progress, not a how-to for the general public.
The data set that was snapshotted was fairly small.
Two successful runs, even with no failures, is not a very big sample set. :)
I didn’t use Postgres’s point-in-time recovery (PITR) here at all—I just started up the database and let Postgres recover from an apparent crash. Shipping over the few WAL logs from the master collected during the pg_backup run after the snapshot copying is complete would allow a theoretically fully reliable recovery to be made, not just a practically non-failing recovery as I did above.

So there’s more work to be done to prove this technique viable in production for a mission-critical database, but it’s a promising start worth further investigation. It shows that there is a way to back up a database across multiple EBS volumes without adding noticeably to its I/O load by utilizing the Amazon EBS data store’s snapshotting and letting a separate EC2 server offload the I/O of backups or anything else we want to do with the data.

Slony: Cascading Subscriptions

2010-01-28T00:00:00+00:00

Sometime you run into a situation where you need to replicate one dataset to many machines in multiple datacenters, with different costs associated with sending to each (either real costs as in bandwidth, or virtual costs as in the amount of time it takes to transmit to each machine). Defining a Slony cluster to handle this is easy, as you can specify the topology and paths taken to replicate any changes.

Basic topology:

Data center A, with machines A1, A2, A3, and A4.
Data center B, with machines B1, B2, B3, and B4.
Data center C, with machines C1, C2, C3, and C4.

Figure 1: Non-cascaded slony replication nodes/pathways.

Node A1 is the master, which propagates its changes to all other machines. In the simple setup, A1 would push all of its changes to each node, however if data centers B and C have high costs associated with transfer to the nodes, you end up transferring 4x the data needed for each data center. (We are assuming that traffic on the local subnet at each data center is cheap and fast.)

The basic idea then, is to push the changes only once to each datacenter, and let the “master” machine in the data center push the changes out to the others in the data center. This reduces traffic from the master to each datacenter, plus removes any other associated costs associated with pushing to every node.

Figure 2: Cascaded slony replication nodes/pathways

Let’s look at an example configuration:

cluster_init.sh:
    #!/bin/bash

    # admin node definitions and other slony-related information are
    # stored in our preamble file.  This will define the $PREAMBLE
    # environment variable that contains basic information common to all
    # Slony-related scripts, such as slony cluster name, the nodes
    # present, and how to reach them to install slony, etc.

    . slony_preamble.sh

    slonik <<EOF
    $PREAMBLE

    init cluster ( id = 1, comment = 'A1' );

    store node (id=2,  comment = 'A2', event node=1);
    store node (id=3,  comment = 'A3', event node=1);
    store node (id=4,  comment = 'A4', event node=1);
    store node (id=5,  comment = 'B1', event node=1);
    store node (id=6,  comment = 'B2', event node=1);
    store node (id=7,  comment = 'B3', event node=1);
    store node (id=8,  comment = 'B4', event node=1);
    store node (id=9,  comment = 'C1', event node=1);
    store node (id=10, comment = 'C2', event node=1);
    store node (id=11, comment = 'C3', event node=1);
    store node (id=12, comment = 'C4', event node=1);

    # pathways from A1 -> A2, A3, A4 and back
    store path (server = 1, client = 2, conninfo = 'dbname=data host=node2.datacenter-a.com');
    store path (server = 1, client = 3, conninfo = 'dbname=data host=node3.datacenter-a.com');
    store path (server = 1, client = 4, conninfo = 'dbname=data host=node4.datacenter-a.com');
    store path (server = 2, client = 1, conninfo = 'dbname=data host=node1.datacenter-a.com');
    store path (server = 3, client = 1, conninfo = 'dbname=data host=node1.datacenter-a.com');
    store path (server = 4, client = 1, conninfo = 'dbname=data host=node1.datacenter-a.com');

    # pathway from A1 -> B1 and back
    store path (server = 1, client = 5, conninfo = 'dbname=data host=node1.datacenter-b.com');
    store path (server = 5, client = 1, conninfo = 'dbname=data host=node1.datacenter-a.com');

    # pathways from B1 -> B2, B3, B4 and back
    store path (server = 5, client = 6, conninfo = 'dbname=data host=node2.datacenter-b.com');
    store path (server = 5, client = 7, conninfo = 'dbname=data host=node3.datacenter-b.com');
    store path (server = 5, client = 8, conninfo = 'dbname=data host=node4.datacenter-b.com');
    store path (server = 6, client = 5, conninfo = 'dbname=data host=node1.datacenter-b.com');
    store path (server = 7, client = 5, conninfo = 'dbname=data host=node1.datacenter-b.com');
    store path (server = 8, client = 5, conninfo = 'dbname=data host=node1.datacenter-b.com');

    # pathway from A1 -> C1 and back
    store path (server = 1, client = 9, conninfo = 'dbname=data host=node1.datacenter-c.com');
    store path (server = 9, client = 1, conninfo = 'dbname=data host=node1.datacenter-a.com');

    # pathways from C1 -> C2, C3, C4 and back
    store path (server = 9, client = 10, conninfo = 'dbname=data host=node2.datacenter-c.com');
    store path (server = 9, client = 11, conninfo = 'dbname=data host=node3.datacenter-c.com');
    store path (server = 9, client = 12, conninfo = 'dbname=data host=node4.datacenter-c.com');
    store path (server = 10, client = 9, conninfo = 'dbname=data host=node1.datacenter-c.com');
    store path (server = 11, client = 9, conninfo = 'dbname=data host=node1.datacenter-c.com');
    store path (server = 12, client = 9, conninfo = 'dbname=data host=node1.datacenter-c.com');

    EOF

As you can see in the initialization script, we’re defining the basic topology for the cluster. We’re defining each individual node, and the paths that slony will use to communicate events and other status. Since slony needs to communicate status both ways, we need to define the paths for each node’s edge both ways. In particular, we’ve defined pathways from A1 to each of the other A nodes, A1 to B1 and C1, and B1 and C1 to each of their respective nodes.

Now it’s a matter of defining the replication sets and describing the subscriptions for each. We will use something like the following for our script:

cluster_define_set1.sh:
    #!/bin/bash

    # reusing our standard cluster information
    . slony_preamble.sh

    slonik <<EOF
    $PREAMBLE

    create set ( id = 1, origin = 1, comment = 'set 1' );

    set add table ( set id = 1, origin = 1, id = 1, fully qualified name = 'public.table1');
    set add table ( set id = 1, origin = 1, id = 2, fully qualified name = 'public.table2');
    set add table ( set id = 1, origin = 1, id = 3, fully qualified name = 'public.table3');

    EOF

Here we’ve defined the tables that we want replicated from A1 to the entire cluster; there is nothing specific to this particular scenario that we need to consider.

cluster_subscribe_set1.sh:
    #!/bin/bash

    # reusing our standard cluster information
    . slony_preamble.sh

    slonik <<EOF
    $PREAMBLE

    # define our forwarding subscriptions (i.e., A1 -> B1, C1)
    subscribe set ( id = 1, provider = 1, receiver = 5, forward = yes);
    subscribe set ( id = 1, provider = 1, receiver = 9, forward = yes);

    # define the subscriptions for each of the datacenter sets
    # A1 -> A2, A3, A4
    subscribe set ( id = 1, provider = 1, receiver = 2, forward = no);
    subscribe set ( id = 1, provider = 1, receiver = 3, forward = no);
    subscribe set ( id = 1, provider = 1, receiver = 4, forward = no);

    # B1 -> B2, B3, B4
    subscribe set ( id = 1, provider = 5, receiver = 6, forward = no);
    subscribe set ( id = 1, provider = 5, receiver = 7, forward = no);
    subscribe set ( id = 1, provider = 5, receiver = 8, forward = no);

    # C1 -> C2, C3, C4
    subscribe set ( id = 1, provider = 9, receiver = 10, forward = no);
    subscribe set ( id = 1, provider = 9, receiver = 11, forward = no);
    subscribe set ( id = 1, provider = 9, receiver = 12, forward = no);

    EOF

The key points here are that you specify the provider nodes and the receiver nodes to specify how the particular replication occurs. For the subscription to any cascade point (i.e., B1 and C1), you need to have the ‘forward = yes’ parameter to ensure that the events properly cascade to the sub-nodes. In any of the other nodes’ subscription, you should set ‘forward = no’.

In actual deployment of this setup, you would want to wait for the subscription from A1 -> B1 and A1 -> C1 to complete successfully before subscribing the sub-nodes. Additionally, this solution assumes high availability between nodes and does not address failure of particular machines; in particular, A1, B1, and C1 are key to maintaining the full replication.

Common Topics in Scalability

2010-01-04T00:00:00+00:00

It rarely makes sense for a startup business to tackle scalability questions from the outset, because it raises the cost and complexity of development and operational support, while solving problems that a start-up business doesn’t typically yet need solved (i.e. handling tons of users). From a cost-effectiveness perspective, it makes sense to solve these problems when the need is actually evident.

That said, systems can be designed with scalability in mind such that the system easily lends itself to incremental changes that scale up its capacity.

The basic principles and techniques that typically come up in scalability discussions:

horizontal scalability: a particular component is “horizontally scalable” if you can deploy redundant instances of that component in parallel; this is the ultimate scalability win, as it means you can readily add raw processing power for that component at low cost.
vertical scalability: the practice of increasing a single component’s power/capacity to improve performance/throughput is referred to as “vertically scaling” that component. From the layperson’s perspective, this is the most easily-understood technique, as it effectively amounts to buying a faster server, adding RAM to an existing server, etc.
caching: caching can be about raw speed, but is more important from a scalability perspective; caches reduce the overall work a system needs to do for any given request, and can simplify the overall request to entirely eliminate the overhead of multiple-component involvement (for instance, a cache in the application layer can potentially eliminate any calls to the database for a given request)

Horizontal scalability brings the biggest win in terms of potential system capacity, as it allows you to handle increased demand (whether that demand is visitors, orders, etc.) through the simple means of adding more servers to your infrastructure. Virtualization and, its logical result, cloud hosting make this kind of infrastructure expansion simpler and often more cost-effective than ever before.

Some examples of horizontal scalability:

Running your appserver (Rails, Django, etc.) on multiple servers, with a load balancer distributing traffic more-or-less evenly across those servers
- The entire application tier is scaled horizontally and can expand/contract as needed.
- Session management becomes an issue; sessions either get moved to a shared component like the database, or server affinity techniques are used with the load balancer such that a single client/user always hits the same application server, so sessions can be file-based on each app server.
- The database likely becomes the bottleneck for system capacity and performance.
Master/slave database replication, with multiple slave databases fronted by a load balancer distributing database reads across the slaves.
- Database reads, which in typical webapps account for the bulk of database queries, can be spread across multiple servers and are thus scaled horizontally.
- The total number of slave databases is likely limited by the capacity of the master; each additional slave adds some overhead to the master, so diminishing returns eventually kick in.

The ease with which a given component can be scaled horizontally largely comes down to how it manages state, or, in other words: how it manages the data that it works with.

Application servers are generally designed to be “stateless”, which effectively means that the response for a given request is not dependent on a previous request (though the idea of the “session” is obviously a big exception here). Due to this stateless nature, it’s usually cheap and easy to run your application server in a horizontally-scalable configuration.

By contrast, relational databases are all about state: the entire point of the database is to act as the arbiter of state, maintain that data on disk, and answer questions about that data. We typically expect the database to tell the truth and give a consistent answer about each piece of data. The consistency expectation leads to the need for each piece of data to have one and only one canonical location, which means it cannot be scaled across multiple servers. You can scale copies of data across multiple servers (as done in master/slave replication), but the True Value for a bit of state has to live in one place. Therefore, master databases are generally cut off from the glories of horizontal scalability. (Note: “sharding” offers a way to scale writes, but it doesn’t make your database literally horizontally scalable; any given datum still has one canonical location which is limited to vertical scalability per usual).

Enter caching. When you cannot horizontally scale a given component, you can instead store and reuse copies of the results of that component’s operations, to reduce the overall work done by that component. Most modern application server frameworks provide a variety of helpful caching tools right out of the box, and a good cache strategy can squeeze a great deal of throughput out of a simple architecture. However, caching is bigger than the app tier, as the examples show:

HTTP caching: HTTP clients (i.e. web browsers) can cache the resources they request from your system, if you give them the proper instructions on how to do so (via various HTTP headers). At a minimum, browsers ought to be able to cache the images, CSS, and JavaScript files that make up your site’s large scale visual design, which means they don’t need to request those files repeatedly.
HTTP caching redux: HTTP caching reverse proxies (Varnish, Squid, etc.) can sit between your web/app tier and your users’ browsers, and can cache resources from your site based on HTTP headers and other configurable aspects; when users request a resource, they hit the HTTP reverse proxy first, and if the resource is available in the cache, the cached version is used. This means the user gets a faster response and your actual application does less work.
Page caching: By caching full copies of your dynamically-generated resources (web pages), your system can see enormous scalability gains. This can fall logically under HTTP caching or under application-tier caching or somewhere between; the important point is to consider the idea that you cache an entire dynamic page in some manner, as it brings both such big performance wins and potential design complexities/constraints.
Application tier caching: using simple file-based caches, or scalable, shared, distributed cache layers like memcached or redis, your application can cache the results of expensive queries, frequently-used operations/widgets, etc. and thus reuse the fruits of its own labors; this can reduce the computational cost for handling any given request and thus improve both raw request speed and overall throughput (scalability).
Database replication: though not typically referred to as “caching”, the classic master/slave database replication strategy is effectively a very fancy cache. By pushing out copies of data at near-real-time, this lets your master database server do less work while still giving your application servers highly accurate results.
Controlled denormalization: within your database, you can use triggers and such to manage denormalized data, allowing frequently-used or expensive calculations to be cached as part of the database schema. This allows the application to rely upon such calculations at lower cost. Materialized views fit within this category.

These caching examples/categories vary in complexity of implementation, but they share a common principle: increase system capacity by reusing the system’s work. The use of caching naturally involves trade-offs, but in the simple case, a straightforward expiry-based cache can have a dramatic impact on performance. A sluggish webapp can get a second wind by wrapping the most common database-driven components within a timed cache. For new development, for which your caching strategy can factor in at design time, caching can yield great performance/scalability with extremely high accuracy. In particular, the refresh-on-write strategy (in which the code paths in your app responsible for changing state are also responsible for updating related caches) can be a huge scalability win (this was exactly the strategy we used—with great results—for Backcountry.com’s SteepandCheap site in fall of 2007 and the initial launch of their product Q&A features in early 2008).

Ideally, a good caching strategy does not merely reuse the results of earlier work, but in fact cuts out calls to other services; good caching in the application tier may mean that most requests do not need to involve the database at all; good HTTP caching techniques cut down the number of requests that involve your application server. And so on.

Beyond these topics, scalability benefits can frequently come from:

query optimization: poor database usage can result in slow page load times, excess I/O, and can overload the database unnecessarily; query optimization can have a significant impact there. This is rather like a corollary to vertically scaling your database: your database scales better relative to the hardware because its demands on the hardware are reduced.
data management optimization: big scalability gains can often come from revising the structure of the data involved at the scalability pinch points; for instance, having a single inventory count for a given SKU is a scalability bottleneck compared to having an inventory item record per 1 inventory count per sku. The former results in lock contention in high-order-volume situations, while the latter can minimize such locking and prevent one user’s order from blocking another.
application database usage: related to query optimization is the issue of how the application structures its queries. In an age where people increasingly interact with their datasources through an object-relational mapper (ORM), one commonly finds cases in which the application issues per-record queries within a loop, meaning that a set of N records yields N+1 queries (1 query for the initial set, 1 query per iteration). This kind of database usage brings unnecessary I/O overhead and leads to sluggish app performance; rewrite the queries or tweak the ORM as necessary such that the first query loads all the data, or that a second query fetches all the extra data for all members of the first query’s result set (ORMs often will do the second strategy for you if you let them know what you want).

The unfortunately-named “NoSQL” movement has a lot of exciting possibilities for systems that need to handle large volumes of data, or scale out to very high write volumes (I find both Cassandra and HBase to be particularly interesting). However, unless you know from the outset that your dataset size or write volume will be pushing the limits of the traditional relational database, or that you need distributed operations for availability purposes, the NoSQL offerings are quite possibly counterproductive.

These solutions typically offer some particular advantage but with a trade-off of some kind; for small businesses with a rapidly-evolving sense of self and the problem space, the traditional RDBMS brings a huge amount of flexibility in how one works with data, as well as a well-understood operational paradigm (for high availability, reliability, backups, etc.). The “Big Data” benefits of NoSQL probably don’t apply for such systems and the ease with which RDBMSes handle arbitrarily-complex queries allow the growing business/system to develop iteratively and arrive at a better understanding of the true underlying needs. The trade-offs and such are beyond the scope of these musings, so I just won’t give them further consideration here; however, the NoSQL ecosystem is good to know about and certainly can factor into scalability discussions depending on your use cases. Several of them may in fact fit well within a caching strategy, rather than as an authoritative datasource.

Here are some general rules of thumb I would recommend if you want to be ready to scale up when the need arises (as is the whole point of these ramblings):

Write well-organized code: Maintain separation of concerns in your application code, maximize code reuse, and keep your classes, modules, functions, etc. limited in their scope. This keeps each individual code piece simple, maximizes the ease with which things like caching can be introduced, and minimizes the code paths that have to be changed in order to optimize the performance of any given component.
Keep your canonical data normalized: This could probably be stated instead as “model your data relationally”, but that assumes an RDBMS, which isn’t necessarily your only datastore. In any case, normalized data is easier to manage over the life of a system, and just as better-organized code is more-easily optimized for performance/scalability than poorly-organized code, well-organized data is more-easily rearranged or cached, etc. Introduce denormalization in a targeted, needs-based manner, but don’t let that denormalization ever be considered part of your canonical dataset; denormalized data is broken data that you should be able to throw out and rebuild at any time.
Avoid session usage: Sessions are handy, but they in fact violate the statelessness of the application server, introduce one point of complexity (though a manageable point) for horizontal scaling, potentially introduce database scaling problems (if you’re using database-backed sessions), etc. If the state matters enough to keep it from one request to the next, consider modeling it and storing it properly in the database. For state that just can’t go in the database and you just can’t live without…
Use the client: The majority of clients for the majority of webapps are web browsers. Those web browsers typically have storage (cookies for now, and more options coming) and frequently have programming capacity (JavaScript). Use it! In particular, cookies can take the role of the user session, and potentially eliminate (or at least reduce) the need for server-side session state. Work done by the client is work not done by your servers.
Think RESTfully: While you don’t necessarily need to literally design a RESTful application, be mindful of the RESTful design principles and keep your URLs well-designed. Critically, don’t allow the same URL to front wildly-varying resources; such resources are the bane of page/HTTP caching.
Be mindful of user-specific resources: Any resource, document, etc. that you serve with user (or session) specific information therein is a resource that is potentially more difficult to cache effectively. Consider crafting your URLs such that the URL is per-user, if that can fit your problem-space effectively (this allows HTTP or page-level caching to still be an option). If common resources (like a “home page”) need to show per-user details, consider using the client (cookies, JavaScript) to encapsulate the user-specific stuff such that the common resource itself is identical for all users (and thus also remains open to HTTP or page-level caching).