Interchange Redis Sessions

2026-06-16T00:00:00+00:00

Photo by Josh Ausborne, 2022.

Interchange¹ now supports storing user sessions in Redis², an in-memory data store, as a core feature of Interchange 5.12. Sessions are where Interchange keeps client state that makes a catalog operate as a continuous experience–the shopping cart, form values, login status, and other data that persist from one request to the next. How and where that state is stored have a direct impact on a busy catalog’s performance, and Redis gives developers a fast, purpose-built option that sits alongside the storage backends Interchange has always offered.

Session storage in Interchange

By default, Interchange writes each session to a file on disk, and it has long been able to store sessions in a relational database instead by way of Vend::SessionDB, configured with the SessionType DBI and SessionDB directives. The database approach is the natural choice once a catalog is served by more than one Interchange server, since every server needs to read and write the same pool of sessions.

Relational databases certainly can function in this capacity, but a high-traffic session table is not where they shine. Every page view reads, writes, and locks a session row, and that constant churn competes with database performance for the application. Redis is built precisely for this kind of small, hot, frequently updated key/value data, and that makes it a compelling choice for session storage.

Using Redis for sessions

Enabling Redis sessions takes two directives in your catalog’s catalog.cfg. Set SessionType to Redis, and point SessionDB at the address and port of your Redis server:

Variable REDIS_SERVER 127.0.0.1:6379
SessionType Redis
SessionDB   __REDIS_SERVER__

That is the whole of the required configuration. Behind the scenes, the new Vend::SessionRedis module ties Interchange’s session storage to Redis through the standard Redis CPAN module³, so Redis and that module must be installed and reachable from your Interchange host.

Redis sessions also work optionally with the session compression recently introduced.⁴ If you set SessionDBCompression to a viable compression algorithm, session data is compressed on the way into Redis and decompressed on the way out, exactly as it is for database sessions:

SessionDBCompression Zstd

More lists and MoreDBTable

Some catalogs use Interchange’s MoreDB feature to cache search-result sets (“more” lists) into the session database rather than writing to files on disk. This feature is often leveraged for the same reason that SessionDB is used: to support a distributed, load-balanced server environment. MoreDB relies on a real database table, and Redis–having no support in Interchange’s database abstraction layer–cannot provide one. To bridge that gap, there is a new MoreDBTable directive that tells MoreDB which relational table to use for more lists, independent of where sessions themselves live:

MoreDB yes
ifdef REDIS_SERVER
   include dbconf/__SESSIONDB_ENGINE__/sessions.__SESSIONDB_ENGINE__.__MOREDB_EXT__
   MoreDBTable sessions
endif

When you run Redis sessions together with MoreDB, MoreDBTable is required. Without Redis it remains optional: MoreDB falls back to SessionDB just as it always has. The strap demo ships ready-made schema definitions for MySQL, PostgreSQL, and SQLite, so the supporting table is created with the right column types.

Locking and a friendlier welcome for bots

Concurrent requests for the same session have to be serialized so two clicks don’t trample each other’s data. For this release, the RDBMS session backend was reworked to lock session records with proper database transactions rather than the old approach of inserting and deleting lock rows. Redis gets its own locking scheme built on Redis’s atomic SETNX (“set if not exists”) operation, exposed through a new has_lock() method. The mechanism is general enough that any other storage system able to offer the same primitive–Memcached, for example–could adopt it.

Redis sessions add one more practical refinement. Automated crawlers generate an enormous number of “hit and run” sessions that are created once and never used again, and on a database-backed catalog they pile up until something prunes them. Vend::SessionRedis instead gives every brand-new session a fixed 30-minute expiration. A human user comfortably makes a second request within that window, at which point the session is promoted to the full lifetime set by your SessionExpire directive. A bot that never returns simply expires on its own quickly, limiting data bloat, and Redis reclaims the memory without any housekeeping on your part.

Wrapping up

Redis sessions give Interchange developers a fast, horizontally scalable place to keep session state, with optional compression, a robust locking model, and built-in resistance to bot-driven session bloat–all configured with a couple of lines in catalog.cfg. As with the rest of Interchange, the implementation lives in core and is open for the community to extend; the has_lock() interface in particular invites support for additional storage backends.

References

Database Design: Using Documents

2022-03-09T00:00:00+00:00

Using documents in relational databases is increasingly popular. This technique can be practical and efficient when used in fitting circumstances.

Example

Let’s start with an example. Imagine we are scraping web sites for external URLs and store them in a table. We’ll have the web sites table to store the scrape timestamp and another table to store all of the references.

CREATE TABLE web_sites (
    web_site_domain text NOT NULL,
    last_scraped_at timestamptz,

    PRIMARY KEY (web_site_domain)
);

CREATE TABLE refs (
    web_site_domain text NOT NULL,
    ref_location text NOT NULL,
    link_url text NOT NULL,

    PRIMARY KEY (web_site_domain, ref_location, link_url),
    FOREIGN KEY (web_site_domain) REFERENCES web_sites
);

We do not need to bother adding an id to the web_sites table, because we assume there won’t be too many of them. The domain is small and more practical to use as an identifier. If you are curious about advantages of using natural keys, see my previous article.

Normalized Tables

There may be many thousands of unique URLs for a single web site and other web sites may refer to the same URLs. To try to minimize the storage, we can keep the locations and the URLs in separate tables, give them integer identifiers, and have another table for the many-to-many relations.

CREATE TABLE locations (
    location_id bigint NOT NULL GENERATED ALWAYS AS IDENTITY,
    web_site_domain text NOT NULL,
    ref_location text NOT NULL,

    PRIMARY KEY (location_id),
    UNIQUE (web_site_domain, ref_location),
    FOREIGN KEY (web_site_domain) REFERENCES web_sites
);

CREATE TABLE links (
    link_id bigint NOT NULL GENERATED ALWAYS AS IDENTITY,
    link_url text NOT NULL,

    PRIMARY KEY (link_id),
    UNIQUE (link_url)
);

CREATE TABLE locations_links (
    location_id bigint NOT NULL,
    link_id bigint NOT NULL,

    PRIMARY KEY (location_id, link_id),
    FOREIGN KEY (location_id) REFERENCES locations,
    FOREIGN KEY (link_id) REFERENCES links
);

The idea in here is to keep our biggest table narrow. It’d pay off to refer to the locations and links just with integer identifiers when we’ll have very many of them.

Table Sizes

We’ll have many web sites and many URLs, so our lookup tables would be big and the relation table would be even bigger. That is going to be a major problem because of many reasons, one of them being space efficiency. Narrow tables with many rows are not very space efficient especially on Postgres where there’s 24 bytes per-row overhead. To demonstrate it being significant, let’s add one row to each of the tables to see the sizes compared with the overhead.

INSERT INTO web_sites (web_site_domain)
VALUES ('example.com');

INSERT INTO refs (web_site_domain, ref_location, link_url)
VALUES ('example.com', '/source', 'http://example.net/target.html');

INSERT INTO locations (web_site_domain, ref_location)
VALUES ('example.com', '/source');

INSERT INTO links (link_url)
VALUES ('http://example.net/example.html');

INSERT INTO locations_links
VALUES (1, 1);

SELECT table_name, tuple_len, (100.0 * 24 / tuple_len)::int AS overhead_perc
FROM information_schema.tables, LATERAL pgstattuple(table_name)
WHERE table_schema = 'public';

   table_name    | tuple_len | overhead_perc
-----------------+-----------+---------------
 web_sites       |        36 |            67
 refs            |        75 |            32
 locations       |        52 |            46
 links           |        64 |            38
 locations_links |        40 |            60

With just one row and such short domains and URLs, we wouldn’t be gaining anything by having the links on a lookup table. In fact, it’s a lot less efficient. Although, the situation may change with many rows.

Composite Types

We can combine multiple fields in a column to avoid the overhead of many rows in narrow tables. Postgres has good support for composite types and arrays over them. They are useful if you need strict data type checking.

CREATE TYPE web_site_page AS (
    ref_location text,
    link_urls text[]
);

ALTER TABLE web_sites ADD COLUMN pages web_site_page[];

UPDATE web_sites
SET pages = ARRAY[ROW(ref_location, ARRAY[link_url])::web_site_page]
FROM refs;

SELECT pg_column_size(pages) FROM web_sites;

 pg_column_size
----------------
            117

As you see, the new column is not exactly small because composite types and arrays come with their own overheads. Still it could be much better when a web site has many references.

JSON Document Column

The composite types are not very easy to work with. We can store the pages in another format. JSON is the most popular one nowadays. There are multiple options to store JSON in a column in Postgres. Let’s start with the text-based data type.

ALTER TABLE web_sites ADD COLUMN pages_json json;

UPDATE web_sites SET pages_json = to_json(pages);

SELECT pg_column_size(pages_json) FROM web_sites;

 pg_column_size
----------------
             76

What is perhaps also surprising is the JSON being smaller than the array over composite type. This is because Postgres data types are a bit wasteful because of alignment and storing oids of the enclosed data types repeatedly.

Binary JSON

Another JSON data type, jsonb is available in Postgres. It is a binary structure that allows faster access to the values inside the document.

ALTER TABLE web_sites ADD COLUMN pages_jsonb jsonb;

UPDATE web_sites SET pages_jsonb = to_jsonb(pages);

SELECT pg_column_size(pages_jsonb) FROM web_sites;

 pg_column_size
----------------
             98

As you see the text-based JSON is still smaller. This is due to the binary offsets stored in the binary structure for faster access to the fields. The text-based JSON also compresses better.

Size Comparison

Very simple single row data shows the text-based json as the smallest, binary jsonb a bit larger, and the composite type to be the largest. However, the differences would be a lot smaller with more realistic sample data with many items inside the documents. I generated some data and gathered these results (the sizes excluding the indexes).

    model         |   size
------------------|---------
single table      |  436 MiB
normalized tables |  387 MiB
composite type    |  324 MiB
json              |  318 MiB
jsonb             |  320 MiB

You may expect the normalized tables to be the smallest as otherwise the metadata is stored repeatedly in the columns. Though, the results reveal it’s the other way around. This is because of the per-row overhead in Postgres. It becomes noticeable in narrow tables as in this example.

It’s also useful to notice that the most of the size is occupied by the TOAST tables when the data is stored in a single column. TOAST is a mechanism in Postgres that kicks in when a column is large enough which would often be the case when you design your tables this way. In this case, the table excluding the TOASTed part is pretty small which would help any query that doesn’t touch the large column.

Usability

What matters more than size is the usability of having it all in a single column. It is really practical to send and receive this column as a whole all the way from the database to the frontend. Compare this with dealing with lots of small objects used with the normalized tables.

It’s also possible in Postgres to index the document columns to allow efficient searches in them. There are many options to do so with advantages and disadvantages over each other. Though this post is getting long. I’ll leave indexing to another one.

Database Design: Using Composite Keys

2021-05-20T00:00:00+00:00

Photo by Chuttersnap

Whether to use single-column or composite keys is another long-debated subject of database design. I previously wrote to support using natural keys and here I want to make good arguments for using composite keys.

Single-column vs. Composite

Single-column keys are widely used nowadays. I wouldn’t be surprised if many developers today don’t even think database design with composite keys is possible, even though they were essential in the beginning. Relational databases make no assumption that the keys must be composed of a single column.

Let’s see the composite keys with the corporate database example again. First, we’d need departments and employees:

CREATE TABLE departments (
  department_id text NOT NULL,
  department_location text NOT NULL,

  PRIMARY KEY (department_id)
);

CREATE TABLE employees (
  username text NOT NULL,
  department_id text NOT NULL,

  PRIMARY KEY (username),
  FOREIGN KEY (department_id) REFERENCES departments
);

Then our database grows, and we need to split the departments into multiple teams. Here’s what they’ll look like:

| id | department  | team           | members |
| -- | ----------- | -------------- | ------- |
|  1 | sysadmin    | infrastructure |       5 |
|  2 | sysadmin    | internal_tools |       3 |
|  3 | development | internal_tools |       4 |
|  4 | development | web_site       |       8 |

As you noticed there are 2 teams named internal_tools, so we cannot use this as the primary key column. We can add a surrogate auto-increment column to use as the primary key, or make the department and team the primary key. Let’s go with the surrogate key option first to demonstrate the problem:

CREATE TABLE teams (
  team_id int NOT NULL GENERATED ALWAYS AS IDENTITY,
  department_id text NOT NULL,
  team_name text NOT NULL
  team_members int NOT NULL,

  PRIMARY KEY (team_id),
  UNIQUE (department_id, team_name),
  FOREIGN KEY (department_id) REFERENCES departments
);

As you noticed, we used the surrogate column as the primary key, and added an additional unique index to ensure the team name to be unique in with the department. Now, let’s relate the employees with the teams:

ALTER TABLE employees
  ADD COLUMN team_id int NOT NULL,
  ADD FOREIGN KEY (team_id) REFERENCES teams;

Now, we know both the department and the team of an employee, but the problem is that they can point to inconsistent rows. For example, I can INSERT myself as an employee:

| username | department_id | team_id |
| -------- | ------------- | ------- |
| hasegeli | sysadmin      |       3 |

team_id 3 is in development department, so now you’d never know if I am in the sysadmin or development department. This is a very common source of data integrity problems in the databases. Applications have no good option to handle this. They would typically crash or sometimes show the employee in one department and sometimes in the other.

You cannot easily add a constraint to the database to prevent this. The best option would be to remove the department_id when the team_id is added as we know the department of the team anyway, but this option is expensive and not always possible, for example when the team_id can be NULL.

Relations with Composite Keys

Now, let’s create the teams table with a composite key:

DROP TABLE teams CASCADE;

CREATE TABLE teams (
  department_id text NOT NULL,
  team_id text NOT NULL,
  team_members int NOT NULL,

  PRIMARY KEY (department_id, team_id),
  FOREIGN KEY (department_id) REFERENCES departments
);

ALTER TABLE employees
  ALTER COLUMN team_id TYPE text,
  ADD FOREIGN KEY (department_id, team_id) REFERENCES teams;

With this method, we ensure data integrity and don’t need to disturb the existing users of the employees table while adding the teams. They can still reliably use the department_id column. We can also set the team_id as NULL and still maintain the data integrity.

Ease of Change

As we have already seen, one of the benefits of using composite keys is to respond to database model changes easier and without compromising data integrity. This becomes relevant in many real world scenarios. For example, let’s add the employees to rooms which must belong to the same department:

CREATE TABLE rooms (
  department_id text NOT NULL,
  room_id text NOT NULL,
  room_location text NOT NULL,

  PRIMARY KEY (department_id, room_id),
  FOREIGN KEY (department_id) REFERENCES departments
);

ALTER TABLE employees
  ADD COLUMN room_id text,
  ADD FOREIGN KEY (department_id, room_id) REFERENCES rooms;

Nothing special needed to be done in here, and it is often easy to respond to more complicated change requests. Composite keys play well with database constraints. For example, let’s add an employee rank which needs to be unique in every department:

ALTER TABLE employees
  ADD COLUMN employee_rank int,
  ADD CHECK (employee_rank > 0),
  ADD UNIQUE (department_id, employee_rank);

This was so easy because we have the department_id in here. Now, let’s imagine a more complicated change request. We need to divide the departments into sections:

CREATE TABLE sections (
  department_id text NOT NULL,
  section_id text NOT NULL,
  section_location text NOT NULL,

  PRIMARY KEY (department_id, section_id),
  FOREIGN KEY (department_id) REFERENCES departments
);

ALTER TABLE teams
  ADD COLUMN section_id text NOT NULL,
  DROP CONSTRAINT teams_pkey CASCADE,
  ADD PRIMARY KEY (department_id, section_id, team_id),
  ADD FOREIGN KEY (epartment_id, section_id) REFERENCES sections;

ALTER TABLE employees
  ADD COLUMN section_id text NOT NULL,
  ADD FOREIGN KEY (department_id, section_id, team_id) REFERENCES teams;

As you see, we can make this happen with minimal impact to the users of the database. As a final practice, let’s reserve employee ranks between 1 and 10 for the main section of every department:

ALTER TABLE employees
  ADD CHECK (section_id = 'main' OR employee_rank > 10);

Querying

Another advantage of using composite keys is to have more possibility for joining of tables. For example using the tables we created we can join employees to sections without using the teams tables. This would not be possible if we had used single-column keys everywhere.

Join conditions get complicated with composite keys. The USING clause helps. To demonstrate, let’s join all of the tables we created so far:

SELECT *
  FROM employees
    JOIN deparments USING (department_id)
    JOIN sections USING (department_id, section_id)
    JOIN rooms USING (department_id, room_id)
    JOIN teams USING (department_id, section_id, team_id);

The USING clause only works if you name the columns the same on all tables. I recommend doing so.

Another advantage of USING clause is to eliminate duplicate columns on the result set. If you run this query, you would not see the department_id column repeated 5 times.

Performance Considerations

One disadvantage of using composite keys is to store more data on tables as references. You would also need more space for the indexes as the reference columns often need to be indexed. However storage is the cheapest of resources, and the performance advantages easily outweigh the extra storage.

The main performance advantage of using composite keys is eliminating the need for many joins as mentioned before. However, when you do need to join many tables, the query planner would have many different paths. It’s the query planners’ strong suit to find the best join order. Composite keys allow them to come up with better plans in many scenarios. To demonstrate this, let’s get our join-all-tables query and add some WHERE conditions:

SELECT *
  FROM employees
    JOIN deparments USING (department_id)
    JOIN sections USING (department_id, section_id)
    JOIN rooms USING (department_id, room_id);
    JOIN teams USING (department_id, section_id, team_id)
  WHERE username LIKE 'a%' AND
    department_location LIKE 'b%' AND
    section_location LIKE 'c%' AND
    room_location LIKE 'd%';
    team_members > 3;

Now we added 5 conditions using 5 columns on 5 different tables. The query planner can estimate which conditions are more selective and plan to join the tables from the smaller one to bigger one.

Database Design: Using Natural Keys

2021-03-15T00:00:00+00:00

Whether to use natural or surrogate keys is a long-debated subject of database design. I am a fan of using natural keys. I think there are even more compelling reasons to use natural keys in databases as the systems grow more complex and interdependent.

Natural or Surrogate

Let’s start by what we mean by natural. It’s not trivial to define this. In today’s world of APIs, someone’s surrogate key is another’s natural key. Wikipedia defines natural keys as “a type of unique key in a database formed of attributes that exist and are used in the external world outside the database”. This makes it clear that the keys we get from APIs are our natural keys. But how about the ones generated by us to be used in the external world?

When applications expose the keys on the URLs and APIs, others start relying on them. This is where our choices become important. When all those different applications generate their own keys instead of using the keys they got from other places, life becomes difficult for no reason.

Let’s elaborate with an example corporate database where employees are identified by their usernames and the departments with their domains. So our data would look like this:

| department        | username  | job                     |
| ----------------- | --------- | ----------------------- |
| sysadm.corp-x.com | hasegeli  | Database Administrator  |
| sysadm.corp-x.com | john      | System Administrator    |
| dep1.corp-x.com   | jane      | Developer               |

When we design this using surrogate keys, it’d look like this:

CREATE TABLE departments (
  id int NOT NULL GENERATED ALWAYS AS IDENTITY,
  domain text NOT NULL,

  PRIMARY KEY (id),
  UNIQUE (domain)
);

CREATE TABLE employees (
  id int NOT NULL GENERATED ALWAYS AS IDENTITY,
  username text NOT NULL,

  PRIMARY KEY (id),
  UNIQUE (username)
);

CREATE TABLE department_employees (
  id int NOT NULL GENERATED ALWAYS AS IDENTITY,
  department_id int NOT NULL,
  employee_id int NOT NULL,
  job text NOT NULL,

  PRIMARY KEY (id),
  UNIQUE (department_id, employee_id),
  FOREIGN KEY (department_id) REFERENCES departments (id),
  FOREIGN KEY (employee_id) REFERENCES employees (id)
);

Then, we would have to make a query joining them all together:

SELECT d.domain, e.username, de.job
FROM department_employees de
JOIN departments d ON de.department_id = d.id
JOIN employees e ON de.employee_id = e.id;

Instead, we could just have it as we need it on a single table:

CREATE TABLE department_employees (
  domain text NOT NULL,
  username text NOT NULL,
  job text NOT NULL,

  PRIMARY KEY (domain, username)
);

And, we could just query it:

SELECT domain, username, job
FROM department_employees;

Good Examples of Natural Keys

A primary key must be always available (NOT NULL) and unique. It should also be compact and easy to represent. Here are some good examples of natural keys:

Keys acquired from various APIs
Usernames
Domains
Email addresses
ISO country codes
IATA airport codes
Dates

I think easy representation matters for unchanged use among many systems. For that reason, I don’t think names with spaces, symbols, or non-ASCII Unicode letters are good keys.

UUIDs and hashes/digests like MD5 and SHA are also popular as primary keys. I don’t think they are a good choice as surrogate keys. Auto-incrementing IDs are easier to use, generate, store, predict — although, when they are defined in the external world and we have to store them, there’s nothing wrong with using them as the primary keys.

Another consideration when selecting keys is their stability. Things that can change are not good keys. This is not only because it is more work to change the keys everywhere, but because different parts of the system cannot rely on them.

The context matters to evaluate a key’s stability. For example, a phone number is not a good key for the users table, because users may change their phone number. However, it is perfect for the phone numbers table.

Also, I don’t think we should discard some useful keys just because they can change in an edge case. It is a good trade-off to allow the rare change rather than to complicate the system with a surrogate key.

Enterprise Architecture

I don’t know what you understand from the word “enterprise” but I have heard people advocating streamlined, unpractical solutions in the name of “enterprise” for a long time. I have seen people using layers of layers in complex frameworks and ORMs with caches and buffers, managers and supervisors, factories and workers, locks and semaphores, queues and consumers, all in the name of “enterprise architecture”.

Many of those frameworks have a rule like requiring all database tables to have a single-column auto-incremented primary key named “id”. Such a rule is easy to enforce and follow but it’s limiting. It would not be practical, maybe not even possible to use natural keys as the primary keys within such framework.

Even though I am writing this post to support using natural keys, strong conventions built around surrogate keys do certainly help developing complex systems in the “enterprise” environments. They help to keep systems consistent when the developers are more motivated to write code than to read what’s there, while working until they get a better offer from another company.

Performance Consideration

Performance is often an argument in favour of surrogate keys. Surrogate keys can be small auto-incrementing integers which would keep the indexes small, next branches easy to predict, so well performing.

Surrogate keys would take a bit of extra space since otherwise we don’t need to have them at all, but the space savings of using them as foreign keys would easily outweigh this. However, using surrogate keys may require the queries to have more tables joined together.

Some relational databases maintain the tables indexed by the primary key. Such systems are more affected by the size of the primary key. On PostgreSQL all indexes are secondary including the primary key, therefore the primary key is not any more important than the other indexes on the table.

Composite Keys

Natural keys are often unique together with other keys. This forces the database designer to use multiple columns as keys. Such design has many advantages, which I may cover in future blog posts.

Python concurrency: asyncio for threading users

2020-10-05T00:00:00+00:00

Photo by Adrian Schwarz

You’ve probably heard this classic software engineering mantra:

Concurrency is hard.

The undeniable fact is that an entire category of software bugs, known for being elusive and frustrating to reproduce, is gated behind the introduction of concurrency to a project. Race conditions, mutual exclusion, deadlock, and starvation, to name a few.

Most programming languages with concurrency features ship with some or all of the classical concurrency primitives: threads, locks, events, semaphores, mutexes, thread-safe queues, and so on. While practically any concurrency problem can be solved with this toolkit, let me share a relevant life mantra:

Just because you can doesn’t mean you should.

In the case of Python, you have access to a standard library alternative to threading, which factors out many of the trickier parts of concurrent programming: asyncio. Many existing applications of Python threads can be replaced by asyncio coroutines, potentially eliminating many of the difficulties of concurrency.

Understanding the differences between asyncio and threading can help you make informed choices about which to apply and when, so let’s take a closer look.

The Python GIL

Any discussion of Python concurrency should mention Python’s GIL, or Global Interpreter Lock. The GIL ensures that only one thread may be executing Python code at a time. A. Jesse Jiryu Davis writes a succinct description of how the GIL affects Python threads:

One thread runs Python, while N others sleep or await I/O.

Once a thread has acquired the GIL, there are two ways it can release the GIL for another thread to acquire:

Reaching a point in the script where it is sleeping, ending, waiting for I/O, or executing native code which explicitly releases the GIL. For example, calling time.sleep(0) forces a thread to release the GIL without ending the thread.
After executing for some amount of time and/or instructions, a thread may be forced to release the GIL for another thread to have a turn. Interrupting a thread in this way is preemption.

A Contrived Example of Thread Preemption

Here I will set up a concurrency toy to demonstrate some characteristics of Python threading. One thread will increment a list of numbers in a for loop. The other thread will, roughly once per second, check the consistency of the list (all numbers are equal) and print a message. Observant readers can probably imagine a better non-threading solution to this problem, but allow me to entertain you.

All posted results are from running the examples in Python 3.8.6 in a Docker container on a laptop. Your results may vary.

Here is the naive approach, without any attempt at synchronization:

import threading
import time

SIZE = 100000


class Counter():
  def __init__(self):
    self.values = [0] * SIZE

  def count(self):
    while True:
      for i in range(SIZE):
        self.values[i] += 1

  def heartbeat(self):
    t0 = time.monotonic()

    while True:
      time.sleep(1.0)

      # Check for consistency.
      for i in range(SIZE):
        assert self.values[i] == self.values[0], f'Value at index {i} is inconsistent'

      now = time.monotonic()
      print(f'All values are {self.values[0]} at +{now - t0:.5f}s')


if __name__ == '__main__':
  counter = Counter()

  t_count = threading.Thread(target=counter.count, daemon=True)
  t_count.start()

  t_heartbeat = threading.Thread(target=counter.heartbeat, daemon=True)
  t_heartbeat.start()

  print('Press Ctrl+C to end')

  # Wait for Ctrl+C or heartbeat crash.
  t_heartbeat.join()

If you run this code a few times, you’ll notice that you might get a few heartbeat messages (if you’re lucky), then the script crashes because the consistency check failed. Because the counter thread can be preempted at any time, this can include between value increments in the for loop. Similarly, the heartbeat thread can be preempted in the middle of its consistency check.

Threads can be very rude when left alone. We can prevent threads racing for access to shared state by using a Lock, which can only be held by one thread at a time:

class Counter():
  def __init__(self):
    self.values_lock = threading.Lock()
    self.values = [0] * SIZE

  def count(self):
    while True:
      with self.values_lock:
        for i in range(SIZE):
          self.values[i] += 1

  def heartbeat(self):
    t0 = time.monotonic()

    while True:
      time.sleep(1.0)

      # Check for consistency.
      with self.values_lock:
        for i in range(SIZE):
          assert self.values[i] == self.values[0], f'Value at index {i} is inconsistent'

      now = time.monotonic()
      print(f'All values are {self.values[0]} at +{now - t0:.5f}s')

All values are 693 at +9.12154s
All values are 797 at +10.60691s
All values are 877 at +11.83219s
All values are 957 at +13.04729s
All values are 1036 at +14.27122s
All values are 1157 at +16.15205s

When you run this code you may notice that the consistency check never fails, but the timing of the heartbeat message is not as consistent as we would like. In fact, it may be several seconds before you see a heartbeat message. There is a lot of timing jitter, which is deviation from the expected time interval. There is also a lot of drift, which is accumulated jitter.

This is happening because there is no guarantee that the counter thread will release the GIL while values_lock is available, or that values_lock won’t be acquired again by the counter thread immediately after releasing it.

When multiple threads are waiting to acquire a Lock, the order in which they are woken up is not defined. We might expect or sometimes observe that the Lock will be acquired first come, first served, or first in, first out, but Python makes no guarantees.

Remembering that we can explicitly release the GIL, we cleverly try doing so immediately before the counter thread acquires values_lock to give the heartbeat thread a chance to win the race:

def count(self):
  while True:
    time.sleep(0)  # Explicitly yield the GIL.
    with self.values_lock:
      for i in range(SIZE):
        self.values[i] += 1

All values are 114 at +1.41245s
All values are 206 at +2.57230s
All values are 315 at +4.05285s
All values are 432 at +5.75999s
All values are 514 at +6.94572s
All values are 662 at +9.05388s

This is much better, but there is still enough jitter and drift that we might not even be able to adequately compensate for the drift by reducing the sleep interval automatically. We’ve applied all of our knowledge of threading and this is the best we can do, within the constraints of the example.

All of this may seem like a lot to keep up with, and it is. Even if you are a threading master and craft perfectly safe and reasonably performant Python threading code, it is likely to be relatively difficult and expensive to maintain. There’s got to be a better way.

Concurrency with asyncio

The asyncio approach to Python concurrency is relatively new. Its integration with the language has changed over the course of Python development, but it appears to be largely stable and useful as of Python 3.8. Instead of using Python threads to run instructions concurrently, asyncio uses an event loop to schedule instructions on the main thread.

Contrasted with threads, asyncio coroutines may never be interrupted unless they explicitly yield the thread with async or await keywords. However, there is no guarantee that saying async or await will yield the thread to another task.

The asyncio library is intended to be used for I/O-bound applications such as high performance network servers, which spend much of their time waiting for the OS to send or receive data on a file descriptor or socket. However, as we will see when applying asyncio to our toy concurrency example, it can be applied to otherwise pure and isolated Python code too.

This example requires Python 3.7 for the asyncio.run() method.

import asyncio

SIZE = 100000


class Counter():
  def __init__(self):
    self.values = [0] * SIZE

  async def count(self):
    while True:
      await asyncio.sleep(0)  # Explicitly yield to other coroutines.
      for i in range(SIZE):
        self.values[i] += 1

  async def heartbeat(self):
    loop = asyncio.get_event_loop()
    t0 = loop.time()

    while True:
      await asyncio.sleep(1.0)

      # Check for consistency.
      for i in range(SIZE):
        assert self.values[i] == self.values[0]

      now = loop.time()
      print(f'All values are {self.values[0]} at +{now - t0:.5f}s')


async def main():
    counter = Counter()

    tasks = map(asyncio.create_task, [counter.count(), counter.heartbeat()])
    await asyncio.wait(tasks, return_when=asyncio.FIRST_COMPLETED)


if __name__ == '__main__':
    asyncio.run(main())

All values are 84 at +1.04261s
All values are 169 at +2.08948s
All values are 254 at +3.13599s
All values are 326 at +4.18760s
All values are 400 at +5.24720s
All values are 473 at +6.30465s

This timing is now in the ballpark where we can compensate for drift by automatically adjusting the heartbeat sleep interval. There is no more racing for the GIL. Instead the counter coroutine explicitly awaits whenever the state is consistent. It may not be interrupted by another coroutine at any other time, but it can still be interrupted by another thread or a signal.

If we imagine that the asyncio.sleep(0) incantation is actually awaiting data on a file descriptor or network socket saturated with data, this example is immediately relevant to the intended use of asyncio. The pattern of running top-level coroutines with asyncio.wait(), among other asyncio task scheduling tools, can potentially replace many existing uses of daemon threads for background tasks.

A caveat of asyncio is the consequence of one of its advantages: While a coroutine is running, it may never be interrupted by another coroutine unless it explicitly gives up control. One slow or malformed coroutine can freeze your entire application, where daemon threads would keep preempting and bypassing the blockage (at cost of some context-switching overhead and developer sanity). If you must run blocking code concurrently you can run it in an executor to decouple it from the asyncio event loop, but understand that the usual thread synchronization problems may apply.

Just because asyncio never preempts doesn’t mean you will never need to synchronize access to shared state. The library does include threading-like synchronization primitives (which are not thread-safe), but the need for them should be the exception, not the norm.

Conclusion

If you’re interested in using asyncio, I urge you to explore its interfaces further. We’ve only scratched the surface in these examples. If you are shoehorning asyncio into an existing application or want to use it with a library or framework that relies on threads, know there are asyncio interfaces for safely adding coroutines from a thread.

Happy hacking!

Extensible Binary Encoding with CBOR

2019-03-18T00:00:00+00:00

CBOR is a relatively new IETF draft standard extensible binary data format. Compared to similar formats like MessagePack and BSON, CBOR was developed from the ground up with clear goals:

unambiguous encoding of most common data formats from Internet standards

code compactness for encoder and decoder

no schema description needed

reasonably compact serialization

applicability to constrained and unconstrained applications

good JSON conversion

extensibility

—RFC 7049 Appendix E, Copyright © 2013 IETF Trust, Bormann & Hoffman

In the context of data storage and messaging, most developers can relate to CBOR as a binary drop-in replacement for JSON. While CBOR doesn’t share the human readability of JSON, it can efficiently and unambiguously encode types of data that JSON struggles with. CBOR can also be extended with tags to optimize serialization beyond its standard primitives.

Encoding Binary Data

JSON is a ubiquitous data format for web and beyond, for many good reasons, but encoding blobs of binary data is an area where JSON falters. For example, if you are designing a JSON protocol to wrap the storage or transfer of arbitrary objects, your options are:

Require that all input data can be represented as JSON. When possible this is potentially a reasonable solution, but limits the types of data that can be encoded. Notable exceptions include most popular image encodings, excluding SVG.
Base64 encode any binary data values to a string. This can encode any binary data, but increases the size of the data by a minimum of 1/3, incurs encoding and decoding cost, and requires magic to indicate that the string is Base64 encoded.
Encode the bytes as an array of numbers or a hex string. These are probably not things you should do, but it seemed worth mentioning that these techniques increase the size of the data by anywhere from 2x to 5x and also require magic to indicate that the data is really binary.

With CBOR, binary blobs of any length are supported out of the box and are encoded 1:1. Encoding and decoding CBOR byte strings is extremely fast, even in higher-level languages. In the case of JavaScript, CBOR byte strings can be transcoded to and from the very fast and efficient Uint8Array.

For example, let’s try encoding a simple object with two fields: “name” and “data”. An abstract view of this object:

name: "Strawberry Pie"
data: <00 01 02 03 04 05 06 07 08 09>

A C struct of this raw data would be absolute minimum of 24 bytes.

If you Base64 the data into JSON, your output might look like this:

{"name":"Strawberry Pie","data":"AAECAwQFBgcICQ=="}

Length: 51 bytes.

This illustrates the “magic” problem with Base64 encoding binary data in JSON. Unless you have a JSON schema or special protocol fields, the decoder has no indication that the data in this object needs to be Base64 decoded, because it looks like a string. It is not self-describing.

The CBOR representation of the same input data, as a hex string:

a2646e616d656e5374726177626572727920506965696a7065675f646174614a00010203040506070809

Whoa, there. What do all these numbers and letters mean? I miss my JSON! The Node cbor package has a handy cbor2comment tool to annotate this hex string for us.

  a2                -- Map, 2 pairs
    64              -- String, length: 4
      6e616d65      -- {Key:0}, "name"
    6e              -- String, length: 14
      5374726177626572727920506965 -- {Val:0}, "Strawberry Pie"
    64              -- String, length: 4
      64617461 -- {Key:1}, "data"
    4a              -- Bytes, length: 10
      00010203040506070809 -- {Val:1}, 00010203040506070809

Length: 35 bytes.

Now let’s benchmark JSON against CBOR using real JPEG data. For this test I used a modified version of cbor-js, a library compatible with both Node and browsers, and encoded a 910,226 byte JPEG of strawberry rhubarb pie.

|                           | JSON      | CBOR    |
| :------------------------ | --------: | ------: |
| Median encoding time (ms) | 3.983     | 0.538   |
| Median decoding time (ms) | 3.151     | 0.006   |
| Encoded size (bytes)      | 1,213,676 | 910,262 |

As the numbers show, CBOR is both faster and more concise for this particular data. Also, CBOR pie tastes better.

Food always looks good in pictures!

Optimizing CBOR with Tags

In the case of encoding homogeneous numeric arrays, CBOR encoders can struggle with optimizing the packing of the data. For example, if you have an array of floating point numbers in a higher-level language like Python or JavaScript, the CBOR encoder implementation won’t necessarily determine how many bits are required to encode the numbers, defaulting to the largest available. Additionally, each value in the array will be individually described as a floating point number. This increases the cost and size of the data considerably.

CBOR has an answer to this problem. The Draft Typed Array Tags spec includes tags specifying typed arrays which happen to match JavaScript TypedArray flavors.

For example, let’s say you have a Float32Array with a few values:

const floats = new Float32Array([1.234567, 2.345678, 3.456789]);

JSON has a really funny way of encoding a Float32Array, lookit all those bytes:

{"0":1.2345670461654663,"1":2.3456780910491943,"2":3.456789016723633}

Length: 69 bytes.

If we were to help the encoder by sending it a regular Array it would still be pretty verbose, but the precision we weren’t using is truncated, so the overall length will vary wildly depending on the values in the Array:

[1.234567,2.345678,3.456789]

Length: 28 bytes.

The well-meaning Node cbor library can encode the Float32Array directly, but doesn’t try to optimize for size:

  83                -- Array, 3 items
    fb              -- Float, next 8 bytes
      3ff3c0c960000000 -- [0], 1.2345670461654663
    fb              -- Float, next 8 bytes
      4002c3f2e0000000 -- [1], 2.3456780910491943
    fb              -- Float, next 8 bytes
      400ba78100000000 -- [2], 3.456789016723633

Length: 28 bytes.

Look at all that wasted precision, 64 bits for each 32-bit float, this just won’t do! The CBOR spec allows you to “tag” data for special treatment. According to the list of registered CBOR tags, “IEEE 754 binary32, little endian, Typed Array” is tag 85. The consecutive bytes of the three numbers follow.

  d8                --  next 1 byte
    55              -- Tag #85
      4c            -- Bytes, length: 12
        4b069e3f971f1640083c5d40 -- 4b069e3f971f1640083c5d40

Length: 15 bytes.

The contained 12 byte string is equivalent to the values in the ArrayBuffer underneath our Float32Array.

The CBOR decoder will spit out the tag alongside its associated blob, so our optimization is self-described. Now we need to ensure that a byte string with this tag is correctly converted to the typed array, minding endianness. In a mature JavaScript implementation, this is both easy and very fast. Because we encoded the values of the ArrayBuffer underneath a Float32Array, we can construct a new Float32Array from the ArrayBuffer.

const floats = new Float32Array(bytes);  // assuming platform and bytes are same endianness!

This is a very scalable way to read and write arrays of numeric values into JavaScript.

When to Use CBOR

As King Crimson teaches us, “it doesn’t mean you should / just because you can.”

I’ve found CBOR useful as an alternative to JSON for large (>2kB) chunks of raw data or numeric arrays. In many other cases JSON is equivalent or superior, because most languages have native JSON encoding and decoding in their standard libraries. CBOR encoders are not always so optimized, but the gap is closing with wider adoption. Run your own tests and benchmarks on real data against real libraries before deciding to use CBOR.

Many readers will recognize that schemaful formats such as protocol buffers are an endgame for structured data. If the infrastructure demands make sense for your application, this is also a good way to go. If you’re working with an application that already uses JSON, the difference in development and maintenance costs of porting it to CBOR or protobuf-likes should be measured against the size and performance gains of each approach.

It’s tempting to let numbers alone decide what format to use, but don’t underestimate the value of JSON’s human readability for structured data. The value of human readability diminishes quickly when dealing with large numeric arrays or binary blobs, so once again CBOR is an appealing choice for these data.

Bonked By Basic_auth Because Bcrypt

2016-02-09T00:00:00+00:00

Alligator photo by Random McRandomhead

TL;DR: Don’t use a high bcrypt cost with HTTP basic auth!

Recently we had a client approach us with reports of a slow wiki experience. This was for a MediaWiki we recently installed for them; there were no fancy extensions, and the hardware, the OS, and the Apache web server were solid, perfectly normal choices. I was tasked to dive in and solve this issue.

The first step in any troubleshooting is to verify and duplicate the problem. While the wiki did feel a bit sluggish, it was not as bad as the reports we were getting of taking over 15 seconds to view a page. A side-by-side comparison with a similar wiki seemed a good place to start. I called up the main wiki page on both the client wiki and End Point’s internal wiki. Both were running the latest version of MediaWiki, had the same type of servers (located a similar distance from me), were using the same version of Apache, and had roughly the same server load. While both wiki’s pages had roughly the same amount of content, the client one loaded noticeably slower. It took less than a second for the End Point wiki, and around ten seconds for the client one!

The first possible culprit was MediaWiki itself. Perhaps something was misconfigured there, or some extension was slowing everything down? MediaWiki has good debugging tools. Inside the both wiki’s LocalSettings.php file I turned on debugging temporarily with:

$wgDebugLogFile         = '/tmp/mediawiki.debug';
$wgDebugDBTransactions  = true;
$wgDebugDumpSql         = true;
$wgDebugTimestamps      = true;

I reloaded the page, then commented out the $wgDebugLogFile line to stop it from growing large (the debug output can be quite verbose!). Here’s some snippets from the generated log file:

0.9151   4.2M  Start request GET /wiki/Main_Page
...
[caches] main: SqlBagOStuff, message: SqlBagOStuff, parser: SqlBagOStuff
[caches] LocalisationCache: using store LCStoreDB
0.9266   9.2M  Implicit transaction open enabled.
0.9279   9.2M  Query wikidb (1) (slave): SET /* DatabasePostgres::open  */ client_encoding='UTF8'
0.9282   9.2M  Resource id #127: Transaction state changed from IDLE -> ACTIVE
0.9268   9.2M  Query wikidb (2) (slave): SET /* DatabasePostgres::open  */ datestyle = 'ISO, YMD'
...
0.9587   9.2M  Query wikidb (11) (slave): SELECT /* LCStoreDB::get  */  lc_value  FROM "l10n_cache"   WHERE lc_lang = 'en' AND lc_key = 'deps'  LIMIT 1
0.9573   9.5M  Query wikidb (12) (slave): SELECT /* LCStoreDB::get  */  lc_value  FROM "l10n_cache"   WHERE lc_lang = 'en' AND lc_key = 'list'  LIMIT 1
0.9567  10.8M  Query wikidb (13) (slave): SELECT /* LCStoreDB::get  */  lc_value  FROM "l10n_cache"   WHERE lc_lang = 'en' AND lc_key = 'preload'  LIMIT 1
0.9572  10.8M  Query wikidb (14) (slave): SELECT /* LCStoreDB::get  */  lc_value  FROM "l10n_cache"   WHERE lc_lang = 'en' AND lc_key = 'preload'  LIMIT 1
...
0.9875  21.2M  Query wikidb (195) (slave): SELECT /* LCStoreDB::get Greg */  lc_value  FROM "l10n_cache"   WHERE lc_lang = 'en' AND lc_key = 'messages:accesskey-pt-mycontris'  LIMIT 1
0.9873  21.2M  Query wikidb (196) (slave): SELECT /* LCStoreDB::get Greg */  lc_value  FROM "l10n_cache"   WHERE lc_lang = 'en' AND lc_key = 'messages:tooltip-pt-logout'  LIMIT 1
0.9868  21.2M  Query wikidb (197) (slave): SELECT /* LCStoreDB::get Greg */  lc_value  FROM "l10n_cache"   WHERE lc_lang = 'en' AND lc_key = 'messages:accesskey-pt-logout'  LIMIT 1
0.9883  21.2M  Query wikidb (198) (slave): SELECT /* LCStoreDB::get Greg */  lc_value  FROM "l10n_cache"   WHERE lc_lang = 'en' AND lc_key = 'messages:vector-more-actions'  LIMIT 1

Just to load a simple page, there were 194 SELECT statements! And 137 of those were trying to look in the l10n_cache table, one row at a time. Clearly, there is lots of room for improvement there. Someday, I may even jump in and tackle that. But for now, despite being very inefficient, it is also very fast. Because of the $wgDebugTimestamps, it was easy to compute how much time both wikis spent actually creating the page and sending it back to Apache. In this case, the difference was minimal, which meant MediaWiki was not the culprit.

I then turned my attention to Apache. Perhaps it was compiled differently? Perhaps there was some obscure SSL bug slowing things down for everyone? These were unlikely, but it was worth checking the Apache logs (which were in /var/log/httpd). There are two main logs Apache uses: access and error. The latter revealed nothing at all when I loaded the main wiki page. The access logs looked fairly normal:

85.236.207.120 - greg [19/Jan/2016:12:23:21 -0500] "GET /wiki/Main_Page HTTP/1.1" 200 23558 "-" "Mozilla/5.0 Firefox/43.0"
85.236.207.120 - greg [19/Jan/2016:12:23:22 -0500] "GET /mediawiki/extensions/balloons/js/balloon.config.js HTTP/1.1" 200 4128 "https://wiki.endpoint.com/wiki/Main_Page
" "Mozilla/5.0 Firefox/43.0"
...
85.236.207.120 - greg [19/Jan/2016:12:23:22 -0500] "GET /mediawiki/load.php?debug=false&lang=en&modules=mediawiki.legacy.commonPrint%2Cshared%7Cmediawiki.sectionAnchor%7Cmediawiki.skinning.interface%7Cskins.vector.styles&only=styles&skin=vector HTTP/1.1" 200 58697 "https://wiki.endpoint.com/wiki/Main_Page" "Mozilla/5.0 Firefox/43.0"
85.236.207.120 - greg [19/Jan/2016:12:23:22 -0500] "GET /mediawiki/resources/assets/poweredby_mediawiki_88x31.png HTTP/1.1" 200 3525 "https://wiki.endpoint.com/wiki/Main_Page" "Mozilla/5.0 Firefox/43.0"

Still nothing out of the ordinary. What to do next? When all else fails, go to the system calls. It’s about as close to bare metal as you can easily get on a Linux system. In this case, I decided to run strace on the Apache daemon to see exactly where the time was being spent. As expected, there were a large handful of httpd processes already spawned and waiting for a connection. While there was no way to know which one would field my requests, some shell-fu allowed me to strace them all at once:

## The -u prevents us from picking the parent httpd process, because it is owned by root!
$ strace -o greg.httpd.trace -tt -ff `pgrep -u apache httpd | xargs -n 1 echo -p | xargs`
Process 5148 attached
Process 4848 attached
Process 5656 attached
Process 4948 attached
Process 5149 attached
Process 5148 attached
Process 4858 attached
Process 5657 attached
Process 4852 attached
Process 4853 attached
^CProcess 5148 detached
Process 4848 detached
Process 5656 detached
Process 4948 detached
Process 5149 detached
Process 5148 detached
Process 4858 detached
Process 5657 detached
Process 4852 detached
Process 4853 detached

Looking at one of the output of one of these revealed some important clues:

$ head greg.httpd.trace.4948
13:00:28.799807 read(14, "\27\3\3\2\221\0\0\0\0\0\0\0\1\35-\332\3123(\200\302\"\251'g\256\363b5"..., 8000) = 666
13:00:28.799995 stat("/wiki/htdocs/mediawiki/load.php", {st_mode=S_IFREG|0644, st_size=1755, ...}) = 0
13:00:28.800126 open("/wiki/htpasswd.users", O_RDONLY|O_CLOEXEC) = 15
13:00:28.800176 fstat(15, {st_mode=S_IFREG|0640, st_size=947, ...}) = 0
13:00:28.800204 read(15, "alice:{SHA}jA0EAgMCMEpo4Wa3n/9gy"..., 4096) = 2802
13:00:28.800230 close(15)               = 0
13:00:29.496369 setitimer(ITIMER_PROF, {it_interval={0, 0}, it_value={60, 0}}, NULL) = 0
13:00:29.496863 rt_sigaction(SIGPROF, {0x7fc962da7ab0, [PROF], SA_RESTORER|SA_RESTART, 0x7fc970605670}, {0x7fc962da7ab0, [PROF], SA_RESTORER|SA_RESTART, 0x7fc970605670

Aha! If you look close at those timestamps, you will notice that the time gap from the call to close() and the subsequent setitimer() is quite large at .69 seconds. That’s a long time for Apache to be waiting around for something. The second clue is the file it just opened: “htpasswd.users”. Seeing the top of the file, with the {SHA} in quotes, made me realize the problem—htpasswd files now support bcrypt as an authentication method, and bcrypt is designed to be secure—and slow. Sure enough, the htpasswd file had bcrypt entries with a high cost for the people that were having the most issues with the speed. This is what the file looked like (names and values changed):

alice:{SHA}jA0EAgMCMEpo4Wa3n/9gybBBsDPa
greg:$2y$13$+lE6+EwgtzP0m8K8VQDnYMRDRMf6rNMRZsCzko07QQpskKI9xbb/y9
mallory:$2y$15$ww8Q4HMI1Md51kul2Hiz4ctetPqJ95cmspH8T81JHfqRvmg===rVgn
carol:7RnEKJWc38uEO
bob:$apr1$uKX9Z63CqPOGX4lD1R4yVZsloJyZGf+
jon:$2y$08$SUe3Z8sgEpyDWbWhUUUU5wtVTwlpEdc7QyXOg3e5WBwM4Hu35/OSo1
eve:$apr1$I/hv09PcpU0VfXhyG7ZGaMz7Vhxi1Tm

I recognized the bcrypt format right away ($2y$13$). The people who were complaining the most (e.g. mallory in the example above) about the speed of the wiki had the highest costs, while those with low costs (e.g. jon), and those using something other than bcrypt (everyone else above), were not complaining at all!

The ‘cost’ is the number after the second dollar sign: as you can see, some of them had a cost of 15, which is much more expensive than a cost of 13, which is what my user (“greg”) was using. This was a smoking gun, but one more step was needed for proof. I adjusted the cost of my password to something low using the htpasswd program:

$ htpasswd -B -C 6 /wiki/htpasswd.users greg
New password: 
Re-type new password: 
Updating password for user greg

Voila! The page loaded in a flash. I then changed the cost to 15 and suddenly the wiki was even slower than before—taking upwards of 15 seconds to load the main page of the wiki. Mystery solved. All those high cost bcrypt requests are also not good for the server: not only does it use a lot of CPU, but ends up keeping the Apache daemon tied up waiting for the bcrypt to finish, rather than simply finishing up quickly and going back to the main pool.

You may be asking a few questions at this point, however. Why would htpasswd offer a footgun like this? Why such a radical difference in effect for slightly different costs? Is bcrypt a good practice for a htpasswd file? Let’s attempt to answer those. Before we do, we have to learn a little bit about bcrypt and passwords in general. Some of this is purposefully oversimplified, so be gently in the comments. :)

Passwords themselves are never stored on a server (aka the machine doing the authentication). Instead, the server stores a hash of the password. This is created by what is known as a “one-way” function, that creates a unique fingerprint of your password. If this fingerprint (aka hash) is discovered, there is no direct way to see the password that created it. When you login to a site, it creates a hash of the password you give it, then compares that hash to the one it has stored. Thus, it can verify that you have given it the correct password without actually having to store the password.

For a long time, very simple algorithms were used to create these hashes. However, as computers became more powerful, and as the field of cryptography advanced, it became easier to “crack” these hashes and determine the password that was used to create them. This was an important problem, and one of the solutions that people came up with was the bcrypt algorithm, which makes the computation of the hash very expensive, in terms of computer speed. Furthermore, that speed is adjustable, and determined by the “cost” given at creation time. You may have noticed the -C option I used in the htpasswd example above. That number indicates the number of rounds the algorithm must go through. However, the cost given leads to 2^code rounds, which means that the cost is exponential. In other words, a cost of 13 means that bcrypt runs 2 to the 13th power rounds, or 8,192 rounds. A cost of 14 is 2 to the 14th power, or 16,384 rounds—twice as slow as a cost of 13! A cost of 15 is 32,768 rounds, etc. Thus, one can see why even a cost of 15 would be much slower than a cost of 13.

A web page usually returns more than just the requested HTML. There are commonly images, CSS, and JavaScript that must also be loaded from the webserver to fully render the page. Each of these requests must go through basic auth, and thus get slowed down by bcrypt. This is why even though each basic authentication via bcrypt of 15 only takes a couple of seconds, the entire web page can take much longer.

What encryption options are available for htpasswd program? The bcrypt option was introduced without much fanfare in version 2.4.4 of Apache, which was released on February 25, 2013. So, it’s been around a while. The output of –help shows us that bcrypt is the only secure one, but allows for other legacy ones to be used. Also note that the range of costs for bcrypt range from 4 to 31:

 -m  Force MD5 encryption of the password (default).
 -B  Force bcrypt encryption of the password (very secure).
 -C  Set the computing time used for the bcrypt algorithm
     (higher is more secure but slower, default: 5, valid: 4 to 31).
 -d  Force CRYPT encryption of the password (8 chars max, insecure).
 -s  Force SHA encryption of the password (insecure).
 -p  Do not encrypt the password (plaintext, insecure).

So should you use bcrypt for your htpasswd? Absolutely yes. Even a lower cost bcrypt is incredibly more secure than using MD5, CRYPT, or SHA. A cost of 10 is roughly the same speed as those, but a much, much better choice. You can measure the time it takes to create or update your password via the command-line htpasswd command to get a rough idea of how much impact it will have on your website. You can use the time it takes to run the htpasswd command as rough proxy for the total page load time. Here are some numbers I generated on my local box. Numbers represent the average of a few runs:

Bcrypt cost	htpasswd creation time	Web page load time
10	0.079	5.68 seconds
12	0.268	6.77 seconds
14	0.979	10.78 seconds
16	3.684	25.72 seconds
18	14.683	88.85 seconds
20	58.680	358.80 seconds
22	236.369	1357.82 seconds
31	186,173 seconds (51 hours and 42 minutes!!)	Ah...no

There are times where you really do want a higher bcrypt cost. The basic auth usage in this scenario is really the exception, and not the norm. In most cases, a password will be used to log in to something, and you will either create a persistent connection (e.g. SSH), or a cookie with a temporary token will be issued (e.g. almost every website in the world). In those cases, a few seconds delay are quite acceptable, as it is a rare event.

So why do we even care about passwords so much, especially for something like basic auth and a htpasswd file? After all, if someone can view the contents of the htpasswd file, they can also more than likely view whatever material on the web server it was designed to protect. These days, however, it’s important to view strong hashes such as bcrypt as not just protecting data, but protecting the password as well. Why? Password reuse. It’s very common for people to use the same (or very similar) password on all the sites they visit. The danger is thus not that an attacker can view the file contents protected by the htpasswd file, but that an attacker can use that password on the user’s email accounts, or on other sites the user may have visited and used the same password.

What bcrypt cost should you use? The general answer is to use the highest possible cost you can get away with. Take something with such a high cost that is causes discomfort to the users, then dial it back a tiny bit. Measure it out and see what your server can handle. For general bcrypt use, start with 13, but don’t be afraid to keep going up until it takes a wall clock second or two to run. For basic auth, use something very fast: perhaps 9 or less. Anything that takes over a second to create via htpasswd will slow a site down noticeably!

Memcache Full HTML in Ruby on Rails with Nginx

2015-09-04T00:00:00+00:00

Hi! Steph here, former long-time End Point employee now blogging from afar as a software developer for Pinhole Press. While I’m no longer an employee of End Point, I’m happy to blog and share here.

I recently went down the rabbit hole of figuring out how to cache full HTML pages in memcached and serve those pages via nginx in a Ruby on Rails application, skipping Rails entirely. While troubleshooting this, I could not find much on Google or StackOverflow, except related articles for applying this technique in WordPress.

Here are the steps I went through to get this working:

Replace Page Caching

First, I cloned the page caching gem repository that was taken out of the Rails core on the move from Rails 3 to 4. It writes fully cached pages out to the file system. It can easily be added as a gem to any project, but the decision was made to remove it from the core.

Because of the complexities in cache invalidation across file systems on multiple instances (with load balancing), and the desire to skip a shared/mounted file server, and because the Rails application relies on memcached for standard Rails fragment and view caching throughout the site, the approach was to use memcached for these full HTML pages as well. A small portion of the page (my account and cart info) is modified via JavaScript with information it retrieves from another server.

The following [simplified] changes were made to the actionpack-page_caching gem, to modify where the full page content was stored:

# Cache clearing update:
+ Rails.cache.delete(path)
- File.delete(path) if File.exist?(path)
- File.delete(path + '.gz') if File.exist?(path + '.gz')

# Cache write update:
+ Rails.cache.write(path, content, raw: true)
- FileUtils.makedirs(File.dirname(path))
- File.open(path, 'wb+') { |f| f.write(content) }
- if gzip
-   Zlib::GzipWriter.open(path + '.gz', gzip) { |f| f.write(content) }
- end

# Cache path change:
def page_cache_path(path, extension = nil)
+ path
- page_cache_directory.to_s + page_cache_file(path, extension)
end

See, it’s not that much! The rest of the gem was not modified much, and the interaction from the Rails app to this gem was maintained (via a controller class method :caches_page). The one thing to note above is the raw option passed in the call to write the cache, which forces the content to be served as a raw string.

Step 2: Set up nginx to look for memcached files

Next, I had to set up nginx to serve the HTML from memcached. This was the tricky part (for me). After much experimentation and logging, I finally settled on the following simplified config:

location / {
    set $memcached_key $uri;
    set $memcached_request 1;
    default_type "text/html";

    if $uri ~ "admin") {
      set $memcached_request 0;
    }

    if ($uri ~ "nocache") {
      set $memcached_request 0;
    }

    if ($memcached_request = 1) {
      memcached_pass localhost:11211;
      error_page 404 = /nocache/$uri;
    }
  }

The desired logic here is to look up the memcached pages for all requests. If there is no memcached page, nginx should fallback to serving the standard Rails page with a modified URL (“/nocache/” prepended). Without this URL modification, nginx would get stuck in an infinite loop of looking up all URLs in memcache repeatedly.

Step 3: Setup a Rack::Rewrite rule to lookup /nocache/ pages.

Due to the infinite loop problem, Rails was receiving all requests with /nocache/ prepended to it. A simple solution to handle this was to add a Rack::Rewrite rule to internally rewrite the URL to ignore the /nocache/ fragment, shown below. The nice thing about this change is that if caching is disabled (e.g. on the development server), this rewrite rule won’t affect any requests.

config.middleware.insert_before(Rack::Runtime, Rack::Rewrite) do
  rewrite %r{/nocache/(.*)}, '/$1'
end

Step 4: Cache invalidation: the really hard part, right?

Finally, I had to add cache invalidation throughout the application where the memcache pages needed to expire. There are a few options for this, for example:

Inside a Rails controller after a successful update/commit.
Inside a Rails model, via ActiveRecord callbacks.
Inside a module that decorates an ActiveRecord model via callbacks.

Choose your poison wisely here. A controller makes sense, but in the case where RailsAdmin is utilized for all admin CRUD methods, it’s not much different (IMHO) to extend the controllers as it is to extend the models. I’m a fan of ActiveRecord callbacks, so I went with option 2.

Final Thoughts

Logs were invaluable here. Most importantly, the memcached log was invaluable here to confirm the infinite loop bug. Also, the Rails dev log was naturally helpful once I solved the nginx issue to handle the Rack::Rewrite rule.

One important note here is that this type of caching does not take advantage of expiration via model timestamp cache keys introduced in Rails 4. But, it wouldn’t be able to, because nginx needs a quick lookup on the URL in memcache to serve the file, and we ideally don’t want to hit the database to try to figure out what that key should be.

Interchange Loop Optimization

2015-02-09T00:00:00+00:00

It’s important to understand both how loops work in Interchange and the (very) fundamental differences between interpolating Interchange tag language (ITL) and the special loop tags (typically referred to as [PREFIX-*] in the literature). Absent this sometimes arcane knowledge, it is very easy to get stuck with inefficient loops even with relatively small loop sets. I’ll discuss both the function of loops and interpolation differences between the tag types while working through a [query] example. While all loop tags–[item-list], [loop], [search-list], and [query]–process similarly, it is to [query] where most complex loops will gravitate over time (to optimize the initiation phase of entering the loop) and where we have the most flexibility for coming up with alternative strategies to mitigate sluggish loop-processing.

Loop Processing

All loop tags are container tags in Interchange, meaning they have an open and close tag, and in between is the body. Only inside this body is it valid to define [PREFIX-] tags (notable exception of [PREFIX-quote] for the sql arg of [query]). This is because the [PREFIX-] tags are not true ITL. They are tightly coupled with the structure of the underlying rows of data and they are processed by distinct, optimized regular expressions serially. Outside the context of the row data from a result set, they are meaningless.

Moreover, the body of a loop tag is slurped into a scalar variable (as all bodies of container tags are handled via the ITL parser) and for each row in the record set of the loop, the contents are acted upon according to the [PREFIX-] tags defined within the body. The first important distinction to recognize here is, the per-row action on this scalar is limited to only the [PREFIX-] tags. The action occurring at loop time ignores any embedded ITL.

At the end of each row’s processing, the copy of the body tied to that one row is then concatenated to the results of all previous rows thus processed. For a loop with N rows (assuming no suppression by [if-PREFIX-*] conditionals) that means every instance of ITL originally placed into the loop body is now present N times in the fully assembled body string. Simple example:

[loop list='1 2 3']
[tmp junk][loop-code][/tmp]
[if scratch junk == 2]
I declare [loop-code] to be special!
[else]
Meh. [loop-code] is ordinary.
[/else]
[/if]
[/loop]

Once this result set with N=3 is processed, but before Interchange returns the results, the assembled return looks like the following string:

[tmp junk]1[/tmp]
[if scratch junk == 2]
I declare 1 to be special!
[else]
Meh. 1 is oridinary
[/else]
[/if]

[tmp junk]2[/tmp]
[if scratch junk == 2]
I declare 2 to be special!
[else]
Meh. 2 is oridinary
[/else]
[/if]

[tmp junk]3[/tmp]
[if scratch junk == 2]
I declare 3 to be special!
[else]
Meh. 3 is oridinary
[/else]
[/if]

Some important observations:

It doesn’t take much ITL to turn a loop body into a monster interpolation process. One must consider the complexity of the ITL in the body by a factor of the number of rows (total, or the “ml” matchlimit value).
ITL does nothing to short-circuit action of the [PREFIX-*] tags. Having [PREFIX-param], [PREFIX-calc], etc. inside an ITL [if] means all those loop tags parse regardless of the truth of the if condition.

ITL vs. Loop Tags

ITL maps to routines, both core and user-defined, determined at compile time. They are processed in order of discovery within the string handed to the ::interpolate_html() routine and have varied and complex attributes that must be resolved for each individual tag. Further, for many (if not most) tags, the return value is itself passed through a new call to ::interpolate_html(), acting on all embedded tags, in an action referred to as reparse. There is, relatively speaking, a good deal of overhead in processing through ::interpolate_html(), particularly with reparse potentially spawning off a great many more ::interpolate_html() calls.

Loop tags, by contrast, map to a pre-compiled set of regular expressions. In contrast to crawling the string and acting upon the tags in the order of discovery, each regex in turn is applied globally to the string. The size of the string is limited to the exact size of the single loop body, and there is no analogue to ITL’s reparse. Further, given this processing pattern, a careful observer might have noted that the order of operations can impact the structure. Specifically, tags processed earlier cannot depend on tags processed later. E.g., [PREFIX-param] processes ahead of [PREFIX-pos], and so:

[if-PREFIX-pos 2 eq [PREFIX-param bar]]

will work, but:

[if-PREFIX-param bar eq [PREFIX-pos 2]]

will not. While the above is a somewhat contrived exampled, the impacts of loop tag processing can be more easily seen in an example using [PREFIX-next]:

[PREFIX-next][PREFIX-param baz][/PREFIX-next]
Code I only want to run when baz is false, like this [PREFIX-exec foo][/PREFIX-exec] call

Because [PREFIX-next] is the absolute last loop tag to run, every other loop tag in the block is run before the next condition is checked. All [PREFIX-next] does is suppress the resulting body from the return, unlike Perl’s next, which short-circuits the remaining code in the loop block.

An Optimization Example

As long as you’re familiar with the idiosyncracies of [PREFIX-*] tags, you should make every effort to use them instead of ITL because they are substantially lighter weight and faster to process. A classic case that can yield remarkable performance gains is to directly swap an embedded [perl] or [calc] block with an equivalent [PREFIX-calc] block.

Let’s take a typical query with little consideration given to whether we use loop tags or ITL, not unlike many I’ve seen where the resource has just become unusably slow. This code originally was developed processing 50 records per page view, but the team using it has requested over time to increase that count.

[query
    list=1
    ml=500 [comment]Ouch! That's a big N[/comment]
    sql="
        SELECT *
        FROM transactions
        WHERE status = 'pending'
        ORDER BY order_date DESC
    "
]
Order [sql-param order_number]
[if global DEVELOPMENT]
    Show [sql-exec stats_crunch][sql-param stats][/sql-exec], only of interest to developers
[/if]
Date: [convert-date format="%b %d, %Y at %T"][sql-param order_date][/convert-date]
[if cgi show_inventory]
Inv:
[if cgi show_inventory eq all]
* Shipped: [either][sql-param is_shipped][or]pending[/either]
* Count: [inventory type=shipped sku=[sql-param sku]]
[/if]
* On Hand: [inventory type=onhand sku=[sql-param sku]]
* Sold: [inventory type=sold sku=[sql-param sku]]
* Shipping: [inventory type=shipping sku=[sql-param sku]]
[/if]
Order details:
    <a href="[area
                href=order_view
                form="
                    order=[sql-param order_number]
                    show_status=[either][cgi show_status][or]active[/either]
                    show_inventory=[cgi show_inventory]
                "
            ]">View [sql-param order_number]</a>
[/query]

Considering this block out of context, it doesn’t seem all that unreasonable. However, let’s look at some of the pieces individually and see what can be done.

We use [if] in 3 different circumstances in the block. However, those values they test are static. They don’t change on any iteration. (We are excluding the potential of any other ITL present in the block from changing their values behind the scenes.)
[convert-date] may be convenient, but it is only one of a number of ways to address date formatting. Our database itself almost certainly has date-formatting routines, but one of the benefits of [convert-date] is you could have a mixed format underlying the data and it can make sense out of the date to some degree. So perhaps that’s why the developer has used [convert-date] here.
Good chance that stats_crunch() is pretty complicated and that’s why the developer wrote a catalog or global subroutine to handle it. Since we only want to see it in the development environment, it’d be nice if it only ran when it was needed. Right now, because of ITL happening on reparse, stats_crunch() fires for every row even if we have no intention of using its output.
We need that link to view our order, but on reparse it means ::interpolate_html() has to parse 500 [area] tags along with [either] and [cgi] x 500. All of these tags are lightweight, but parsing numbers are really going to catch up to us here.

Our goal here is to replace any ITL we can with an equivalent use of a loop tag or, absent the ability to remove ITL logically, to wrap that ITL into a subroutine that can itself be called in loop context with [PREFIX-exec]. The first thing I want to address are those [if] and [either] tags, the lowest hanging fruit:

[query
    list=1
    ml=500
    sql="
        <b>SELECT *,
            '[if global DEVELOPMENT]1[/if]' AS is_development,
            [sql-quote][cgi show_inventory][/sql-quote] AS show_inventory,
            COALESCE(is_shipped,'pending') AS show_inventory_shipped</b>
        FROM transactions
        WHERE status = 'pending'
        ORDER BY order_date DESC
    "
]
Order [sql-param order_number]
<b>[if-sql-param is_development]</b>
    Show [sql-exec stats_crunch][sql-param stats][/sql-exec], only of interest to developers
<b>[/if-sql-param]</b>
Date: [convert-date format="%b %d, %Y at %T"][sql-param order_date][/convert-date]
<b>[if-sql-param show_inventory]</b>
Inv:
<b>[if-sql-param show_inventory eq all]</b>
* Shipped: <b>[sql-param show_inventory_shipped]</b>
* Count: [inventory type=shipped sku=[sql-param sku]]
<b>[/if-sql-param]</b>
* On Hand: [inventory type=onhand sku=[sql-param sku]]
* Sold: [inventory type=sold sku=[sql-param sku]]
* Shipping: [inventory type=shipping sku=[sql-param sku]]
<b>[/if-sql-param]</b>
Order details:
    <a href="[area
                href=order_view
                form="
                    order=[sql-param order_number]
                    show_status=[either][cgi show_status][or]active[/either]
                    show_inventory=[cgi show_inventory]
                "
            ]">View [sql-param order_number]</a>
[/query]

By moving those evaluations into the SELECT list of the query, we’ve reduced the number of interpolations to arrive at those static values to 1 or, in the case of the [either] tag, 0 as we’ve offloaded the calculation entirely to the database. If is_shipped could be something perly false but not null, we would have to adjust our field accordingly, but in either case could still be easily managed as a database calculation. Moreover, by swapping in [if-sql-param is_development] for [if global DEVELOPMENT], we have kept stats_crunch() from running at all when in the production environment.

Next, we’ll consider [convert-date]:

Date: [convert-date format="%b %d, %Y at %T"][sql-param order_date][/convert-date]

My first attempt would be to address this similarly to the [if] and [either] conditions, and try to render the formatted date from a database function as an aliased field. However, let’s assume the underlying structure of the data varies and that’s not easily accomplished, and we still want [convert-date]. Luckily, Interchange supports that same tag as a filter, and [PREFIX-filter] is a loop tag:

Date: <b>[sql-filter convert_date."%b %d, %Y at %T"]</b>[sql-param order_date]<b>[/sql-filter]</b>

[PREFIX-filter] is very handy to keep in mind as many transformation tags have a filter wrapper for them. E.g., [currency] -> [PREFIX-filter currency]. And if the one you’re looking at doesn’t, you can build your own, easily.

Now to look at that [inventory] tag. The most direct approach assumes that the code inside [inventory] can be run in Safe, which often it can even if [inventory] is global. However, if [inventory] does run-time un-Safe things (such as creating an object) then it may not be possible. In such a case, we would want to create a global sub, like our hypothetical stats_crunch(), and invoke it via [PREFIX-exec]. However, let us assume we can safely (as it were) invoke it via the $Tag object to demonstrate another potent loop option: [PREFIX-sub].

[if-sql-param show_inventory]
<b>[sql-sub show_inventory]
    my $arg = shift;
    return $Tag->inventory({ type => $arg, sku => $Row->{sku} });
[/sql-sub]</b>
Inv:
[if-sql-param show_inventory eq all]
* Shipped: [sql-param show_inventory_shipped]
* Count: <b>[sql-exec show_inventory]shipped[/sql-exec]</b>
[/if-sql-param]
* On Hand: <b>[sql-exec show_inventory]on_hand[/sql-exec]</b>
* Sold: <b>[sql-exec show_inventory]sold[/sql-exec]</b>
* Shipping: <b>[sql-exec show_inventory]shipping[/sql-exec]</b>
[/if-sql-param]

Let’s go over what this gives us:

[PREFIX-sub] creates an in-line catalog sub that is compiled at the start of processing, before looping actually begins. As such, the [PREFIX-sub] definitions can occur anywhere within the loop body and are then removed from the body to be parsed.
The body of the [PREFIX-exec] is passed to the sub as the first argument. We use that here for our static values to the “type” arg. If we also wanted to access [sql-param sku] from the call, we would have to include that in the body and set up a parser to extract it out of the one (and only) arg we can pass in. Instead, we can reference the $Row hash within the sub body just as we can do when using a [PREFIX-calc], with one minor adjustment to our [query] tag–we have to indicate to [query] we are operating on a row-hash basis instead of the default row-array basis. We do that by adding the hashref arg to the list:

[query
    list=1
    ml=500
    <b>hashref=1</b>

We still have access to the full functionality of [inventory] but we’ve removed the impact of having to parse that tag 2000 times (in the worst-case scenario) if left as ITL in the query body. If we run into Safe issues, that same sub body can either be created as a pre-compiled global sub or, if available, we can set our catalog AllowGlobal in which case catalog subs will no longer run under Safe.

Finally, all we have left to address is [area] and its args which themselves have ITL. I will leverage [PREFIX-sub] again as an easy way to manage the issue:

<b>[sql-sub area_order_view]
    my $show_status = $CGI->{show_status} || 'active';
    return $Tag->area({
        href => 'order_view',
        form => "order=$Row->{order_number}\n"
              . "show_status=$show_status\n"
              . "show_inventory=$CGI->{show_inventory}",
    });
[/sql-sub]</b>
Order details:
    <a href="<b>[sql-exec area_order_view][/sql-exec]</b>">View [sql-param order_number]</a>

By packaging all of [area]’s requirements into the sub body, I can address all of the ITL at once.

So now, let’s put together the entire [query] rewrite to see the final product:

[query
    list=1
    ml=500
    hashref=1
    sql="
        SELECT *,
            '[if global DEVELOPMENT]1[/if]' AS is_development,
            [sql-quote][cgi show_inventory][/sql-quote] AS show_inventory,
            COALESCE(is_shipped,'pending') AS show_inventory_shipped
        FROM transactions
        WHERE status = 'pending'
        ORDER BY order_date DESC
    "
]
Order [sql-param order_number]
[if-sql-param is_development]
    Show [sql-exec stats_crunch][sql-param stats][/sql-exec], only of interest to developers
[/if-sql-param]
Date: [sql-filter convert_date."%b %d, %Y at %T"][sql-param order_date][/sql-filter]
[if-sql-param show_inventory]
[sql-sub show_inventory]
    my $arg = shift;
    return $Tag->inventory({ type => $arg, sku => $Row->{sku} });
[/sql-sub]
Inv:
[if-sql-param show_inventory eq all]
* Shipped: [sql-param show_inventory_shipped]
* Count: [sql-exec show_inventory]shipped[/sql-exec]
[/if-sql-param]
* On Hand: [sql-exec show_inventory]on_hand[/sql-exec]
* Sold: [sql-exec show_inventory]sold[/sql-exec]
* Shipping: [sql-exec show_inventory]shipping[/sql-exec]
[/if-sql-param]
[sql-sub area_order_view]
    my $show_status = $CGI->{show_status} || 'active';
    return $Tag->area({
        href => 'order_view',
        form => "order=$Row->{order_number}\n"
              . "show_status=$show_status\n"
              . "show_inventory=$CGI->{show_inventory}",
    });
[/sql-sub]
Order details:
    <a href="[sql-exec area_order_view][/sql-exec]">View [sql-param order_number]</a>
[/query]

Voila! Our new query body is functionally identical to the original body, though admittedly a little more complicated to set up. However, the trade-off in efficiency is likely to be substantial.

I recently worked on a refactor for a client that was overall very similar to the above example, with a desired N value of 250. The code prior to refactoring took ~70s to complete. Once we had completed the refactor using the same tools as I’ve identified here, we brought down processing time to just under 3s, losing no functionality.

Time taken optimizing Interchange loops will almost always pay dividends.

Rails Performance with Skylight

2014-06-26T00:00:00+00:00

Back at RailsConf, I met a couple of the creators of Skylight.io, a recently launched Ruby on Rails profiler. I was anxious to try it out after having unconvincing experiences with New Relic, but first I had to get through a pretty big upgrade of Rails 2.3 to 4.1 on H2O. I survived the upgrade and moved on to profiling.

Installation of Skylight.io was super simple and the installation screen provided real-time feedback during gem installation and configuration. The web-app had data to share within a minute or so. At the moment, Skylight offers a free month trial to get started, and paid plans after that month. Skylight reports metrics requests (referenced by controller#action) per minute and time per request and allows you to sort results by those metrics combined (Response time x Requests Per Minute = Agony), individually or alphabetically. The Agony-sorted option highlights methods that are candidates for the most impactful changes. One interesting note here is because our application uses full page caching, the requests recorded by Skylight do not include those static cached requests, so the Skylight data is a representation of the work the Rails application is doing to generate non-cached content and apply various writes.

Skylight app screenshot offering various ways to sort requests. Controller names blurred out.

Once you drill down to a specific controller#action method, Skylight provides a waterfall of the various processes, including view rendering and database hits. It highlights potential problem areas where the same query executes repeatedly in one request (n+1). There’s a lot of data and interactivity available in the waterfall view.

Skylight request waterfall screenshot. Table names blurred out

With my recent upgrade from Rails 2.3 to Rails 4.1, I initially chose the simple, happy, and wise path of minimal refactoring, which did not take advantage of improved cache management in Rails 4 or eager loading strategies. Armed with Skylight metrics, I was able to apply a number of changes to improve performance, described below.

Data Model Challenges

Before I go into the performance details, I want to describe the inherent challenge associated with the application’s data model, which combines nesting of listed items and polymorphism. In the diagram below, ItemA, ItemB, ItemC, and ItemD are all Rails models. A model of type ItemA has a list of items (of class ListItem). Each of those list items points to another item via a polymorphic relationship (of class ItemA, ItemB, ItemC, ItemD). The nested referenced item can include further nesting. Nesting is allowed at up to 4 levels and infinite nested loops are not allowed. When the top-level ItemA loads, there are some metrics pulled from the aggregate of all of its nested list items, which requires all nested items to be loaded from the database. Because of this nested data model, one must pay special attention to eager loading in Rails (via the includes() method, or default scope). In some cases, eager loading of all nested items is necessary and in other cases it only becomes a performance burden if the data is not needed. This nested polymorphic data model has created some challenges in terms of performance and cache invalidation.

ItemA
  ListItem => ItemA
    ListItem => ItemA
      ListItem => ItemB
    ListItem => ItemA
      ListItem => ItemB
    ListItem => ItemB
  ListItem => ItemB
  ListItem => ItemC
  ListItem => ItemD

Repeating Queries

Skylight reported a number of (n+1) scenarios. This was a relatively simple improvement with a couple of changes:

updating the default_scope of various models to include associations often included
updating specific queries to use the includes(:some_association) method to eagerly load these associations.

I also found an opportunity via Sunspot, a Solr-based Rails search gem, for eager-loading associations on associations to search results objects. Here are some code examples:

default_scope { includes(:some_association) } # example default scope in model
SomeModel.includes(:some_association).limit(5) # example eager loading associations on query
SomeModel.search(:include => :user) # example Sunspot search with eager loading

These updates proved extremely valuable in terms of minimizing database hits by reducing repeated queries.

Cache Management

One of the problem areas that Skylight highlighted was that many of our writes were taking quite a while. In the Rails 2.3 app, Rails Sweepers were used extensively to perform manual cache expiration after specific actions (e.g. create or update). With the update to Rails 4.1, the code can take advantage of better Rails cache key management as well as Russian Doll caching, eliminating the need for manual cache management. The application still uses full page caching in some instances, so some cache management is required to clear the fully cached pages, but the fragment cache management has improved dramatically.

# Example of cache key based on item only
<% cache(item)  do -%>
# Stuff here
<% end -%>

# Example of cache key based on item and item.user
# Fragment cache will expire when item.user or item is updated
<% cache([item.user,  "listed-item", item]) do -%>
# Stuff here
<% end -%>

Remove unnecessary AJAX Requests

Although this wasn’t a specific issue highlighted by Skylight, I did take the opportunity to investigate where AJAX requests could be reduced. In one case, this meant moving from an eager loading strategy via AJAX to a lazy loading strategy. In another scenario, it meant utilizing Rails render_to_string method which allowed me to return both view content and object data from a JSON post, instead of making two requests that return different data types (JSON followed by HTML).

# Example of render_to_string to return HTML and JSON data from single AJAX request
content = render_to_string("path/to/view.html.erb", :locals => { :item => item })
render :json => { :some_key => item,
                  :content => content,
                  :other_data => some_other_data
                }

Database Management

Because Skylight highlighted a few areas where writes were taking excessively long, I revisited options for more efficient interaction with the database. This included updates such as using update_column instead of update_attribute, which eliminated redundant cache expiration logic. This also included minor updates to minimize the number of updates applied to the database where applicable. Those are both duh! updates to a seasoned Rails developer, but having a profiler point out the most agonizing requests (including updates) forced me to dig in deep on specific actions.

Conclusion

My experience with Skylight has been positive. Skylight is opinionated about what data it provides but what it does present is actionable, compared to other profilers which may present a large and overwhelming amount of information and metrics. Because I’ve provided feedback on Skylight, I know they are continuously making updates and improvements in hopes of improving the service. I definitely suggest trying Skylight out to profile your application.

Speeding Up Saving Millions of ORM Objects in PostgreSQL

2014-04-11T00:00:00+00:00

The Problem

Sometimes you need to generate sample data, like random data for tests. Sometimes you need to generate it with huge amount of code you have in your ORM mappings, just because an architect decided that all the logic needs to be stored in the ORM, and the database should be just a dummy data container. The real reason is not important—the problem is: let’s generate lots of, millions of rows, for a sample table from ORM mappings.

Sometimes the data is read from a file, but due to business logic kept in ORM, you need to load the data from file to ORM and then save the millions of ORM objects to database.

This can be done in many different ways, but here I will concentrate on making that as fast as possible.

I will use PostgreSQL and SQLAlchemy (with psycopg2) for ORM, so all the code will be implemented in Python. I will create a couple of functions, each implementing another solution for saving the data to the database, and I will test them using 10k and 100k of generated ORM objects.

Sample Table

The table I used is quite simple, just a simplified blog post:

CREATE TABLE posts (
  id SERIAL PRIMARY KEY,
  title TEXT NOT NULL,
  body TEXT NOT NULL,
  payload TEXT NOT NULL
);

SQLAlchemy Mapping

I’m using SQLAlchemy for ORM, so I need a mapping, I will use this simple one:

class BlogPost(Base):
    __tablename__ = "posts"

    id = Column(Integer, primary_key=True)
    title = Column(Text)
    body = Column(Text)
    payload = Column(Text)

The payload field is just to make the object bigger, to simulate real life where objects can be much more complicated, and thus slower to save to the database.

Generating Random Object

The main idea for this test is to have a randomly generated object, however what I really check is the database speed, and the whole randomness is used at the client side, so having a randomly generated object doesn’t really matter at this moment. The overhead of a fully random function is the same regardless of the method of saving the data to the database. So instead of randomly generating the object, I will use a static one, with static data, and I will use the function below:

TITLE   = "title"      * 1764
BODY    = "body"       * 1764
PAYLOAD = "dummy data" * 1764

def generate_random_post():
    "Generates a kind of random blog post"
    return BlogPost(title=TITLE, body=BODY, payload=PAYLOAD)

Solution Ideas

Generally there are two main ideas for such a bulk inserting of multiple ORM objects:

Insert them one-by-one with autocommit
Insert them one-by-one in one transaction

Save One By One

This is the simplest way. Usually we don’t save just one object, but instead we save many different objects in one transaction, and making a couple of related changes in multiple transactions is a great way leading to a database with bad data.

For generating millions of unrelated objects this shouldn’t cause data inconsistency, but this is highly inefficient. I’ve seen this multiple times in code: create an object, save it to the database, commit, create another object and so on. It works, but is quite slow. Sometimes it is fast enough, but for the cost of making a very simple change in this algorithm we can make it 10 times faster.

I’ve implemented this algorithm in the function below:

def save_objects_one_by_one(count=MAX_COUNT):
    for i in xrange(1, MAX_COUNT+1):
        post = generate_random_post()
        session.add(post)
        session.commit()

Save All in One Transaction

This solution is as simple as: create objects, save them to the database, commit the transaction at the end, so do everything in one huge transaction.

The implementation differs only by four spaces from the previous one, just run commit() once, after adding all objects:

def save_objects_one_transaction(count=MAX_COUNT):
    for i in xrange(1, MAX_COUNT+1):
        post = generate_random_post()
        session.add(post)
    session.commit()

Time difference

I ran the tests multiple times, truncating the table each time. The average results of saving 10k objects were quite predictable:

Multiple transactions — 268 seconds
One transaction — 25 seconds

The difference is not surprising, the whole table size is 4.8MB, but after each transaction the database needs to write the changes on disk, which slows the procedure a lot.

Copy

So far, I’ve described the most common methods of generating and storing many ORM objects. I was wondering about another, which may seem surprising a little bit at the beginning.

PostgreSQL has a great COPY command which can copy data between a table and a file. The file format is simple: one table row per one file row, fields delimited with a defined delimiter etc. It can be a normal csv or tsv file.

My crazy idea was: how about using the COPY for loading all the generated ORM objects? To do that, I need to serialize them to a text representation, to create a text file with all of them. So I created a simple function, which does that. This function is made outside the BlogPost class, so I don’t need to change the data model.

def serialize_post_to_out_stream(post, out):
    import csv
    writer = csv.writer(out, delimiter="\t", quoting=csv.QUOTE_MINIMAL)
    writer.writerow([post.title, post.body, post.payload])

The function above gets two parameters:

post — the object to be serialized
out — the output stream where the row with the post object will be saved, in Python it is a file-like object, so an object with all the functions a file object has

Here I use a standard csv module, which supports reading and writing csv files. I really don’t want to write my own function for escaping all the possible forms of data I could have—this usually leads to many tricky bugs.

The only thing left is to use the COPY command. I don’t want to create a file with data and load that later; the generated data can be really huge, and creating temporary files can just slow things down. I want to keep the whole procedure in Python, and use pipes for data loading.

I will use the psql program for accessing the PostgreSQL database. Psql has a different command called \COPY, which can read the csv file from psql’s standard input. This can be done using e.g.: cat file.csv | psql database.

To use it in Python, I’m going to use the subprocess module, and create a psql process with stdin=subprocess.PIPE which will give me write access to the pipe psql reads from. The function I’ve implemented is:

def save_objects_using_copy(count=MAX_COUNT):
    import subprocess
    p = subprocess.Popen([
        'psql', 'pgtest', '-U', 'pgtest',
        '-c', '\COPY posts(title, body, payload) FROM STDIN',
        '--set=ON_ERROR_STOP=true'
        ], stdin=subprocess.PIPE
    )
    for i in xrange(1, MAX_COUNT+1):
        post = generate_random_post()
        serialize_post_to_out_stream(post, p.stdin)
    p.stdin.close()

Results

I’ve also tested that on the same database table, truncating the table before running it. After that I’ve also checked this function, and the previous one (with one transaction) on a bigger sample—100k of BlogPost objects.

The results are:

Sample size	Multiple Transactions	One Transaction	COPY
10k	268 s	25 s	5 s
100k	—	262 s	51 s

I haven’t tested the multiple transactions version for 100k sample, as I just didn’t want to wait multiple hours for finishing that (as I run each of the tests multiple times to get more reliable results).

As you can see, the COPY version is the fastest, even 5 times faster than the full ORM version with one huge transaction. This version is also memory friendly, as no matter how many objects you want to generate, it always needs to store one ORM object in memory, and you can destroy it after saving.

The Drawbacks

Of course using psql poses a couple of problems:

you need to have psql available; sometimes that’s not an option
calling psql creates another connection to the database; sometimes that could be a problem
you need to set up a password in ~/.psql file; you cannot provide it in the command line

You could also get the pcycopg2 cursor directly from the SQLAlchemy connection, and then use the copy_from() function, but this method needs to have all the data already prepared in memory, as it reads from a file-like object, e.g. StringIO. This is not a good solution for inserting millions of objects, as they can be quite huge—streaming is much better in this case.

Another solution to this is to write a generator, which is a file like object, and the copy_from() method can read from it directly. This function calls the file’s read() method trying to read 8192 bytes per call. This can be a good idea when you don’t have access to the psql, however due to the overhead for generating the 8192 bytes strings, it should be slowever than the psql version.

The more the merrier? Not always...

2014-02-05T00:00:00+00:00

Recently we were working on an image manipulation function for a customer that used ImageMagick’s convert command. We had spent tons of time working on the program and the associated system that was going to use it. Everything was working fine. The whole system was nice and snappy. Great job everyone!

Until we moved it into production where the page load times were 8 to 10 times slower than in our camps development system …

We all sprang into action to try to figure out why the page load times were so slow now. We reviewed system settings, configuration files for the database and application, and anything else we could think of. We made sure the OS packages, ImageMagick version, and various other things were the same, and everything looked right. The production hardware has twice the RAM and 4 times the number of processors as development does. So what the heck is going on?

To distract ourselves and hope for more insight, we tried to optimize the code a bit and while making it a bit better we were still 6 to 8 times slower than in development. We deactivated the section of the site overnight so we could sleep on it. Luckily this was a new product line so it wasn’t tragic to turn it off.

The next morning while discussing, a co-worker mentioned that the larger number of processors in production could be relevant. At first I was a bit taken aback because this goes against everything we think about more being better. How could it be that having 4 times the number of processors was a bad thing?!

Well, it turns out that ImageMagick is a threaded program and will use as many processors in parallel as it can to accomplish whatever task it is asked to do. In doing this instead of splitting up the convert over 8 processors, it was now splitting it up over 32 processors. The extra work it was taking to manage the work being split between those extra 24 processors actually greatly slowed down the processing work!

Luckily ImageMagick respects an environment variable setting to limit the maximum number of threads it will use. We set the OMP_NUM_THREADS environment variable to 4, re-ran the code, and then it performed as expected in production. It could be set to more or less threads, but we found that was fine and aside from keeping performance high, it keeps this one process from dominating CPU usage while it is running.

It was a very interesting riddle to solve so I figured I’d share it.

Database federation performance showdown

2013-12-19T00:00:00+00:00

Flickr user garryknight

The PostgreSQL Foreign Data Wrapper has gotten a fair bit of attention since its release in PostgreSQL version 9.3. Although it does much the same thing the dblink contrib module has long done, it is simpler to implement for most tasks and reuses the same foreign data wrapper infrastructure employed by several other contrib modules. It allows users to “federate” distinct PostgreSQL databases; that is, it allows them to work in combination as though they were one database. This topic of database federation has interested me for some time—I wrote about it a couple years ago—and when postgres_fdw came out I wanted to see how it compared to the solution I used back then.

First, some background. The key sticking point of database federation that I’m focused on is transaction management. Transactions group a series of steps, so either they all complete in proper sequence, or none of them does. While lots of databases, and other technologies like messaging servers, can handle transactions that involve only one service (one database or one messaging server instance, for example), federation aims to allow transactions to span multiple services. If, for instance, given a transaction involving multiple databases, one database fails to commit, all the other databases in the transaction roll back automatically. See my post linked above for a more detailed example and implementation details. In that post I talked about the Bitronix transaction manager, whose job is to coordinate the different databases and other services in a transaction, and make sure they all commit or roll back correctly, even in the face of system failures and other misbehavior. There are other standalone transaction managers available. I used Bitronix simply because a knowledgeable friend recommended it, and it proved sufficient for the testing I had in mind.

So much for introduction. I wanted to see how Bitronix compared to postgres_fdw, and to get started I took the simple sequence of queries used by default by pgbench, and created a test database with pgbench, and then made three identical copies of it (named, of course, athos, porthos, aramis, and dartagnan—I wasn’t energetic enough to include the apostrophe in the name of the fourth database). The plan was to federate athos and porthos with Bitronix, and aramis and dartagnan with postgres_fdw. More precisely, the pgbench test schema consists of a small set of tables representing a simple banking scenario. In its default benchmark, pgbench selects from, inserts into, and updates these tables with a few simple queries, shown below. Like pgbench, my test script replaces identifiers starting with a “:” character with values selected randomly for each iteration.

UPDATE pgbench_accounts SET abalance = abalance + :delta WHERE aid = :aid;
SELECT abalance FROM pgbench_accounts WHERE aid = :aid;
UPDATE pgbench_tellers SET tbalance = tbalance + :delta WHERE tid = :tid;
UPDATE pgbench_branches SET bbalance = bbalance + :delta WHERE bid = :bid;
INSERT INTO pgbench_history (tid, bid, aid, delta, mtime) VALUES (:tid, :bid, :aid, :delta, CURRENT_TIMESTAMP);

I decided to configure my test as though pgbench’s “accounts” table was in one database, and the “tellers”, “branches”, and “history” tables were in another. For the Bitronix test I can simply connect to both databases and ignore the tables that aren’t applicable, but for testing postgres_fdw I need to set up dartagnan’s pgbench_accounts table as a foreign table in the aramis database, like this:

aramis=# drop table pgbench_accounts;
DROP TABLE
aramis=# create server dartagnan foreign data wrapper postgres_fdw options (dbname 'dartagnan');
CREATE SERVER
aramis=# create user mapping for josh server dartagnan options (user 'josh');
CREATE USER MAPPING
aramis=# create foreign table pgbench_accounts (aid integer not null, bid integer, abalance integer, filler character(84)) server dartagnan;
CREATE FOREIGN TABLE

The test script I wrote has two modes: Bitronix mode, and postgres_fdw mode. For each, it repeats the pgbench test queries a fixed number of times, grouping a certain number of these iterations into a single transaction. It then changes the number of iterations per transaction, and repeats the test. In the end, it gave me the following results, which I found very interesting:

The results show that for small transactions, postgres_fdw performs much better. But when the transactions get large, Bitronix catches up and takes the lead. The graph shows a curve that may be part of an interesting trend, but it didn’t seem worthwhile to test larger numbers of iterations per single transaction, because the larger transactions in the test are already very large compared to typical real-life workloads. It’s difficult to see exactly what’s going on in the center of the graph; here’s a log rescaling of the data to make it clear what the numbers are up to.

All in all, it doesn’t surprise me that postgres_fdw would be faster than Bitronix for small and medium-sized transactions. Being more tightly coupled to PostgreSQL, it has a faster path to get done what it wants to do, and in particular, isn’t restricted to using two-phase commit, which is generally considered slow. I was surprised, however, to see that Bitronix managed to catch up for very large transactions.

Full Page Caching in Interchange 5

2013-10-28T00:00:00+00:00

I recently attended the eCommerce Innovation Conference 2013 with fellow End Pointer Richard Templet and presented on the work End Point has done to develop full-page caching in Interchange 5. The work is in final review currently for inclusion into core Interchange and should provide a roadmap for cache management in Nitesi/Interchange 6.

Parameters of a Caching Solution

In order to identify the full scope of what one means when one says an application uses caching, there are (at least) 3 aspects to define:

Where in the application stack is the cache being applied.
How long until the cache is expired.
What level of user state must it support.

The issue of cache duration is not addressed here as any such formulation will be specific to business requirements and tolerance of stale data. I will state, however, that even very brief cache durations can have an enormous impact on performance, particularly for sites that have a relatively small number of resources that absorb the bulk of the traffic. Cache durations of several minutes to several hours can drop the traffic that makes it to Interchange to a small fraction of the overall requests.

Caching and the Application Stack

Let’s examine an ideal target architecture for a simple implementation of an Interchange store, or stores. Note that we could also introduce load balancing layers to substantially boost capacity through horizontal scaling at multiple points in this stack, and if we did so we’d need to add those to our list to identify the impact of selecting a target cache point.

Browser
Reverse proxy (e.g., Varnish, Pound, nginx)
Web server (e.g., Apache httpd, nginx)
Interchange, with session storage
- Database (e.g., MySQL)
- Other applications (e.g., email server)
- Web services (e.g., payment gateway)

The higher up the stack we can cache, the more scalable our solution becomes. Interchange comes with [timed-build] which can be used to great effect to cache database results in particular, but also potentially other applications that could produce bottlenecks in performance. Moreover, because Interchange assembles the document, this is the last point in the stack that we can (directly, if at all) partially cache a resource. However, having all requests reach Interchange is, itself, a scalability issue that would have to be resolved with either horizontal scaling or pushing our cache farther up the stack.

It’s also possible to produce a static build of assets in the web server’s doc space, keeping requests from reaching Interchange at all. And while responses from a web server will have considerably less overhead and better response times than Interchange, both building and maintaining a static repository of Interchange assets is going to take some effort and, ultimately, will require horizontal scaling to relieve overload.

Our target point for the cache described herein is at the reverse proxy. We want to control our cache using standard cache headers and an nginx configuration that uses the full URL for its cache keys. The reverse proxy is chosen because:

Very fast and efficient at delivering documents.
Low maintenance as we are able to have proxy engine keep its data storage fresh according to document headers.
Assets can now be massively scaled through 3rd-party CDNs requiring no infrastructure investment and maintenance that horizontal scaling solutions lower in the stack would.
Full URL cache keys and cache headers allow us extend our cache all the way to the browser, for those user agents that respect the cache headers.

Core Interchange Support for Full-Page Caching

To provide general support for full-page caching in Interchange, we found it necessary to introduce some core features into Interchange 5. These features are in final review internally and will be committed to core soon.

SuppressCachedCookies

New boolean catalog configuration parameter that, when true, instructs Interchange not to write any cookies in a cacheable response. Cookies are inherently specific to the individual client being served and nginx will refuse to cache a resource containing Set-Cookie headers.

$::Instance->{Volatile}

Value indicates to critical core code what the request’s cache potential is. Three values:

undef - unknown, could be cached, but hasn’t been explicitly identified
true - cannot be cached. Indicates the requested resource is user-dependent and may produce different results for the same URL for different users.
false (other than undef) - explicitly treat as a “can be cached” resource. This setting can be used to reverse override other cache overrides.

The ternary nature of Volatile allows a developer to explicitly control the caching behavior of any given resource if circumstances require an adjustment to the default behavior.

[if-not-volatile]

New container tag whose body will only interpolate if the value of $::Instance->{Volatile} at the time of interpolation is false. Tag is particularly useful for placing settings for cache headers on shared resources (includes files, components, etc.) where the final document may or may not be cacheable.

OutputCookieHook

Catalog configuration parameter that takes the name of a catalog or global sub to execute just prior to Interchange writing its Set-Cookie headers. Setting was inspired by the need to maintain portions of the session on the client via cookies to allow some more stubborn session-dependent resources to be cacheable.

Obstacles to Full-Page Caching in Interchange

Standard coding practices in a typical Interchange catalog interfere with full-page caching in a number of ways, primarily:

Liberal coupling of resources with user sessions
Searches common to all site users (e.g., category lists) generate saved search objects that force session coupling
Heavy reliance on non-RESTful URLs, primarily those generated by the process actionmap

In most circumstances, these practices can be altered to produce fully cacheable resources, and particularly if the most heavily used components of the site are addressed first, such as the home page and category lists.

Catalog Changes to Mitigate Caching Obstacles

Precisely what changes are required depends on the specific coding practices used for the resources in question. However, there are a number of typical usage patterns that will to some degree affect almost all Interchange catalogs.

Use RESTful URLs

Avoid unnecessary dependence on the process actionmap, which is often used liberally precisely because it gives lots of hooks into cool and useful features. Avoid any other use of common URLs that produce varying resources based on session dependence.

Take advantage of writing custom actionmaps, which allow the developer extreme flexibility in URL construction. Because actionmaps make it easy and straightforward to produce unique URLs, they are ideal both for creating cacheable resources and fine-tuning SEO.

Permanent More for Category Lists

By default, search objects which Interchange uses for more lists, are restricted to access from the generating user’s session. This is a safeguard as often search results include personal data for access only to the requestor. However, for features such as category lists, this creates a difficult burden for the developer who wishes to cache the popular resources and whose results are identical across all users.

We can overcome this difficulty by making the search definitions for category lists, or other canned searches, include the permanent more indicator. Permanent more causes all identical searches to share a common search object accessible by the same URLs, and freeing the usual coupling with the session of the search originator.

Address Common Session Variables

There are certain session variables that are often found in page code and can cause a number of difficulties when trying to make a resource cacheable. Start by tracking through the use of each of the following and come up with a strategy to remove the dependencies so that the interpolated code is free of them:

[value]
[data session ___]
[set]/[seti]
Any [if] conditionals that use those respective bases (e.g., [if value], [if session], and [if scratch])

Cacheable Redirects

Any code that issues a redirect must do so consistently with respect to its URL. Any redirect will cache the http code and, if issued conditionally, will force all users accessing the cached resource to also redirect. This practice is seen often in Interchange catalogs, particularly when monitoring pages that are restricted to logged-in users. In summary, it’s OK to cache redirects, but just make sure that a given URL is either always, or never, redirected.

Profile and “Click” Code

It is common practice to define profile and click code in scratch variables. This is particularly true for click code defined with the [button] tag, which while convenient causes the click action to be defined under the hood in scratch space. In order for these event-driven features to work, the resource must compile that code and seed it in the session in anticipation of the user’s next actions. If those resources are cached, those important features are never added to the session as the result of a page load and, so, none of the actions will work.

All use of [button] or [set] to produce click or profile code should be moved into the profile files (typically found in etc/profile.*). There they are added to the Interchange global configuration at compile time and are thus available to all users without regard to the state of their sessions. This is good practice generally since it is often easy (particularly with [button]) to have multiple actions map to the same scratch key. When that happens, a user going through the browser back button can get invalid results on actions taken because the click or profile definitions have changed with respect to the anticipated such actions on the current page.

Set SuppressCachedCookies to Yes

As described earlier, this will tell the Interchange core not to write any cookies if the resource is to be cached.

Define Cache-Control Headers

At any point in the development of the response body, Interchange can be issued a pragma that tells it to treat the response as cacheable. This will interact with the core features described above to ensure that there is no impact on the user session as a result of this request, and put the correct headers in the output stream (as well as keep the cookie headers out).

Invocation looks something like [tag pragma cache_control]max-age=NNN[/tag], where NNN is the number of seconds the cache should persist.

Impact on Session Management

Any resource considered cacheable should a priori have neither impact nor dependence on a session. This must be true if we consider that, once cached, a user will interact with the page—and expect correct behavior—without ever touching Interchange. This introduces some new conditions associated with the session:

The initial user request against a cacheable resource will not generate a session. Why is this so? Apart from the already-noted discrepancy of accessing the resource for a cached vs. live hit, generating a session would necessitate producing the session cookie. Returning that cookie would invalidate the resource as cacheable. Further, one of the significant advantages of a reverse-proxy caching strategy is to provide protection in a DoS attack, and the user agents in such an attack are very unlikely to maintain a session. Thus, if we were failing to produce a cache on initial hits to allow setting a session, all those DoS hits would reach Interchange, and on top of that be churning out session entries on the server.
Session writing is suppressed on any response with a cacheable resource. Interchange must treat the response without any permanence because all accesses of the resource from the cache will never reach Interchange. If the request that produced the cache also wrote that user’s session, it would produce a deviation in behavior between the cached v. live requests.

Overrides on a Cacheable Response

Any action resulting in a POST is considered to imply the necessity of the user initiating the request reaching the session (or the database, or some other permanent storage controlled by the Interchange app). Thus POSTs by default force the Volatile setting to true. However, note this can be overridden by the developer if necessary (e.g., if a DoS hits the home page with a POST rather than the expected GET).

Similarly, any requests passing through either the “process” or “order” actionmap are assumed to require access to the session. “process” will most often be issued as a POST as well, although using “order” with a GET is common.

User State on Cacheable Resources

A big mistake a developer may make when considering full-page caching is to assume an all-or-nothing approach. Trying to compartmentalize an entire catalog into fully cacheable resources would be a daunting task, requiring essentially the construction of a fully client-side application and session management. This is neither realistic nor desirable.

A catalog can gain considerable benefit simply from evaluating those resources which do not require session entanglement at all and starting with them. Without considering users that are logged in or have items in their cart, under most circumstances the home and category list pages should be free from entanglement. With a bit of URL management, the resources can skip the cache when a user is logged in or has items in the cart.

“Read Only” User State

If caching is desirable on resources that cannot be decoupled from session influence, we can expose the necessary parts of the session to the client in the form of cookies and can refactor our document to contain client-side code to manage the session use. Typical examples of this would be personalization for logged in users, or the display of a small cart on all pages. The session data stored in the cookie is controlled exclusively by Interchange and is read-only on the client. Each time the session is accessed and updated, the cookie is re-written to the client.

Management of such a process is relatively easy with modern JavaScript frameworks such as jQuery. As a typical example, one might need to replace the following session-dependent code

[if session logged_in]
  Hi, [value fname]!
[/if]

with client-side management:

<span id="fname_display"></span>

and in the ready() event elsewhere with our session cookie data stored in valuesData:

if (valuesData.fname) {
  $('#fname_display').replaceWith('Hi, ' + valuesData.fname + '!');
}

The OutputCookieHook was developed as a convenient mechanism for constructing the proposed valuesData cookie above, allowing for a subroutine to build the cookie just prior to the core code that constructs the document headers but after any standard actions that would alter the session and would need to be captured in the cookie data.

“Read Write” User State

If state needs are more complex on a particularly popular resource, it may be necessary to allow our state cookie to also be updated from the client. With the tools described here, the developer can either amend the existing cookie, or construct a new one, that captures data input by the client through subsequent requests to cached resources. Once the user issues the next non-cached request, an Autoload subroutine could be constructed to identify that changes have occurred and then sync those changes back to the user’s session. While implementing read-write user state may be challenging, it is possible and has been done at End Point for clients where that need exists.

Hopefully this provides a good idea of how to get started when approaching full-page caching, which is possible in most web app frameworks, and soon will be much easier in Interchange 5 with the new core tools introduced here.

Log Jam: Be careful with local syslog

2013-08-22T00:00:00+00:00

Elephant by Shaun Tarlton

All they really wanted to do is log any query that takes over 10 seconds. Most of their queries are very simple and fast, but the application generates a few complicated queries for some actions. Recording anything that took longer than 10 seconds allowed them to concentrate on optimizing those. Thus, the following line was set in postgresql.conf:

log_min_duration_statement = 10

Log Everything

A little while back, Greg wrote about configuring Postgres to log everything, and the really good reasons to do so. That isn’t what they intended to to here, but is effectively what happened. The integer in log_min_duration_statement represents milliseconds, not seconds. With a 10ms threshold it wasn’t logging everything the database server was doing, but enough that this performance graph happened:

Reconstructed I/O Utilization

That is, admittedly, a fabricated performance plot. But it’s pretty close to what we were seeing at the time. The blue is the fast SAS array where all the Postgres data resides, showing lower than normal utilization before recovering after the configuration change. The maroon behind it is the SATA disk where the OS (and /var/log) resides, not normally all that active, showing 100% utilization and dropping off sharply as soon as we fixed log_min_duration_statement.

It took a few minutes to track down, as we were originally alerted to application performance problems, but once we spotted the disk I/O metrics it didn’t take long to track down the errant postgresql.conf setting. That the disk jumped to 100% with so much log activity isn’t surprising, but the database resides entirely on separate disks. So why did it affect that so much?

syslog() and the surprise on the socket

If you’re used to using syslog to send your log messages off to a separate server, you may be rather surprised by the above. At least I was; by default it’ll use UDP to transmit the messages, so an overloaded log server will simply result in the messages being dropped. Not ideal from a logging perspective, but it keeps things running if there’s a problem on that front. Locally, messages are submitted to a dgram UNIX socket at /dev/log for the syslog process to pick up and save to disk or relay off to an external system.

The AF_UNIX SOCK_DGRAM socket, it turns out, doesn’t behave just like its AF_INET UDP counterpart. Ordering of the datagrams is preserved and, more importantly here, a full buffer will block rather than drop the messages. As a result in the case above, between syslog’s file buffer and the log socket buffer, once the syslog() calls started blocking, each Postgres backend stopped handling traffic until its log messages made it out toward that slow SATA disk.

As of now, this system has the Postgres logs on the faster array, to mitigate it if there’s any logging problems in the future. But if you’re looking at leaning on syslog to help manage high volumes of log entries, just be aware that it doesn’t solve everything.

Little Spree Big Performance Problems

2013-08-05T00:00:00+00:00

Recently I worked on an online food store serving an area with very little infrastructure. As a result, the orders tended to be really big with lots of products.

The website worked in the following environment:

Ruby 1.9.2
Spree 0.60
Heroku, Bamboo stack
PostgreSQL 9.2.4

H12 timeout errors

The performance problems started when we migrated Bamboo to Cedar on Heroku and replaced Thin webserver with Unicorn. We started getting a lot of Heroku Request timeout errors - H12:

The problems happened mostly when logging in to admin dashboard or during the checkout for the certain orders. H12 errors occur when a HTTP request takes longer than 30 seconds to complete. For example, if a Rails app takes 35 seconds to render the page, the HTTP router returns a 503 after 30 seconds and abandons the incomplete Rails request for good. The Rails request will keep working and logging the normal errorless execution. After completion, the request will indefinitely hang in the application dyno.

We started debugging H12: we set Unicorn timeout to 20 seconds to prevent the runaway requests and installed the rack-timeout gem with the timeout of 10 seconds to raise an error on a slow request. It all came down to a trivial database timeout!

The application source has not changed during the transition from Bamboo to Cedar, but apparently Cedar/Unicorn is much more sensitive to the troubled code. Below is the list of performance bottlenecks and solutions in Spree. Some of them exist in version 0.60 only, but a lot of them are still present in Spree 1.x, which means that your application may have them too.

Issue #1: Real-time reports

Let’s take a closer look at the earlier database timeout code. It came from Admin Dashboard and admin/overview_controller.rb.

The “Best Selling variants" report was being calculated real-time right in the web process:

def best_selling_variants
  li = LineItem.includes(:order).
      where("orders.state = 'complete'").
      sum(:quantity, :group => :variant_id, :limit => 5)
  ...
end

Record counts in the database are large enough to crash the application on Heroku with the database timeout:

irb(main):001:0> LineItem.count
=> 162279
irb(main):002:0> Order.count
=> 13027
irb(main):003:0> Variant.count
=> 14418

Other reports on the Dashboard experience the same problem. They would cause the timeout in turns when logging into Admin:

top_grossing_variants
best_selling_taxons
last_five_orders
biggest_spenders

Solution

Fortunately, in Spree 1.x the internal reporting system has been replaced with Jirafe:

If switching to Spree 1.x is not an option, another way is to move the calculation into a background job, using, for example, delayed_job gem and Heroku Scheduler Addon:

task :statistics => :environment do
  Delayed::Job.enqueue StatisticsJob.new
end

Issue #2: Large numbers

It is an established fact that humans eat a lot! Think about an order of a thousand Heineken 6-pack cans…

Or even something like this:

or this:

Spree, both 0.60 and 1.x, proved to have a huge problem if an order has a lot of line items and/or a large quantity of single line items. The potentially dangerous code can be found all over the place. Consider the following example:

class InventoryUnit < ActiveRecord::Base
  def self.destroy_units(order, variant, quantity)
    variant_units = order.inventory_units.group_by(&:variant_id)[variant.id].sort_by(&:state)
    quantity.ceil.times do
      inventory_unit = variant_units.shift
      inventory_unit.destroy
    end
  end
end

Now imagine what will happen with the order from the first screenshot. We have 15100 inventory units for that one. They will be meticulously destroyed from the inventory one by one in a loop after the checkout. This method was born to crash the application!

Solution

A simple mindful refactoring was enough to solve the problem for me. There is no need to call “destroy” in the loop for every single inventory unit because we can use the efficient “destroy_all” method. I’m sure this can be optimized further, but it was enough to get rid of the timeout:

def self.destroy_units(order, variant, quantity)
  variant_units = order.inventory_units.
    group_by(&:variant_id)[variant.id].
    sort_by(&:state)
  variant_units = variant_units.shift(quantity.ceil)
InventoryUnit.
    where(:order_id => order.id,:variant_id => variant.id).
    order('state asc').limit(quantity.ceil).
    destroy_all
end

Issue #3: Real-time emails

All emails in Spree are sent in real-time.

class Order < ActiveRecord::Base
  def finalize!
    ...
    OrderMailer.confirm_email(self).deliver
    ...
  end
end

Why is it bad? Let’s look at the following example from my application:

def confirm_email(order)
  attachments["invoice.pdf"] = {
    'Content-type' => 'application/pdf',
    :content => OrderInvoice.new.to_pdf(order)}

  mail(:subject  => 'Order #' + order.number,
       :from   =>    Spree::Config[:order_from],
       :to => order.email)
end

In my application “confirm_email” was overridden and generated the PDF invoice. The invoice listed all the products in the order and had about 200 lines in it. Again, it lead to the H12 timeout error.

Solution

All emails should be sent in the background rather than in the web request. First, because network operations can take a long time, and second, because generating an email can also be slow. For example, sending in the background can be accomplished using delayed_job gem:

OrderMailer.delay.confirm_email(self)

Issue #4: Lazy-loading

Ecommerce objects are usually complicated with a lot of associations. This is totally fine as long as you eager-load the associations that will be used most with the loaded object later on. In most cases, Spree does not preload associations for its orders. For example, in spree/base_controller.rb:

@order = Order.find_by_number! params[:order_id]

As the result, here is what I see in server console while loading the order display page on the frontend:

…And five more screens like this! The associations of the order — line items, variants and products — generate an additional chain of queries to the database during lazy-loading.

Solution: Eager-loading

If I modify the line from the controller like this…

@order = Order.where(:number => params[:order_id])
         .includes(:line_items => {:product => :taxons, :variant => [:product, :option_values]})
         .includes(:adjustments).first

…SQL queries will shrink down to this:

Eager-loading did the trick. Of course, not everything needs to be loaded eagerly, and the solution varies from case to case. It worked like charm in my case.

Issue #5: Dangerous code all over the place

There a lot of places in the Spree source code that are not optimized for performance. We don’t need to look far for an example, because there is another killer method right near the “destroy_units” one we inspected earlier!

class InventoryUnit < ActiveRecord::Base
  def self.create_units(order, variant, sold, back_order)
    shipment = order.shipments.detect {|shipment| !shipment.shipped? }
    sold.ceil.times {
      order.inventory_units.create(:variant => variant, :state => "sold", :shipment => shipment)
    }
    back_order.ceil.times {
      order.inventory_units.create(:variant => variant, :state => "backordered", :shipment => shipment)
    }
  end
end

And then:

order.line_items.each do |line_item|
  back_order = determine_backorder(order, variant, quantity)
  sold = quantity - back_order
  create_units(order, line_item.variant,  sold, back_order)
end

I received a timeout either in the “sold” or the “backorder” loop, but there wasn’t a time when I didn’t receive a timeout!

Solution

def self.create_units(order, variant, sold, back_order)
  shipment = order.shipments.detect {|shipment| !shipment.shipped? }
  values = sold.ceil.times.to_a.map { "(#{order.id},#{variant.id},'sold',#{shipment.id})" } +
    back_order.ceil.times.to_a.map { "(#{order.id},#{variant.id},'backordered',#{shipment.id})" }
  values.in_groups_of(500, false) do |group|
    InventoryUnit.connection.execute("INSERT INTO inventory_units(order_id, variant_id, state, shipment_id) VALUES #{group.join(',')}")
  end
end

Another example: every line item has the after_create and after_save callbacks. The callback invokes the Order.update! method. Order.update! method calls update_totals method…twice throughout the method.

def update_totals
  self.payment_total = payments.completed.map(&:amount).sum
  self.item_total = line_items.map(&:amount).sum
  self.adjustment_total = adjustments.map(&:amount).sum
  self.total = item_total + adjustment_total
end

Now imagine the order from the second screenshot with a lot of line items. During the checkout “update_totals” will be called each time the line item is saved. Typically, this line would produce a timeout, because the “line_items” association was, of course, not preloaded:

line_items.map(&:amount).sum

I couldn’t list every circumstance like that, because it would require a lot of context to explain the catch, but I can still say many times …

No long-running tasks in the web request!

Be it Spree, Heroku, or any other context, environment or platform, please, never do the following in the web process:

(!!!) Heavy database usage (slow or numerous queries, N+1 queries)
Sending an email
Accessing a remote API (posting to Twitter, querying Flickr, etc.)
Rendering an image or PDF
Heavy computation (computing a fibonacci sequence, etc.)

Say “No” to all these things to ensure a much happier life for your application!

NoSQL benchmark of Cassandra, HBase, MongoDB

2013-03-12T00:00:00+00:00

Benchmark White Paper

We’re excited to have recently worked on an interesting benchmarking project for DataStax, the key company supporting the Cassandra “NoSQL” database for large horizontally-scalable data stores. This was done over the course of about 2 months.

This benchmark compares the performance of MongoDB, HBase, and Cassandra on the widely-used Amazon Web Services (AWS) EC2 cloud instances with local storage in RAID, in configurations ranging from 1-32 database nodes. The software stack included 64-bit Ubuntu 12.04 LTS AMIs, Oracle Java 1.6, and YCSB (Yahoo! Cloud Serving Benchmark) for its lowest-common-denominator NoSQL database performance testing features. Seven different test workloads were used to get a good mix of read, write, modify, and combined scenarios.

Because cloud computing resources are subject to “noisy neighbor” situations of degraded CPU or I/O performance, the tests were run 3 times each on 3 different days, with different EC2 instances to minimize any AWS-related variance.

The project involved some interesting automation challenges for repeatedly spinning up the correct numbers and types of nodes, configuring the node software, running tests, and gathering and collating results data. We kept the AWS costs more reasonable by using Spot Instances for most instances.

You can read more at DataStax’s white paper page and see all the details in the white paper itself.

JavaScript-driven Interactive Highlighting

2013-01-25T00:00:00+00:00

An example of highlighted text, by sergis on Flickr

One project I’ve been involved in for almost two years here at End Point is the H2O project. The Ruby on Rails web application behind H2O serves as a platform for creating, editing, organizing, consuming and sharing course materials that is used by professors and their students.

One of the most interesting UI elements of this project is the requirement to allow highlighting and annotating text interactively. For example, when one reads a physical textbook for a college course, they may highlight and mark it up in various ways with different colors and add annotated text. They may also highlight a section that is particularly important for an upcoming exam, or they may highlight another section with a different color and notes that may be needed for a paper.

The H2O project has required support for digitizing interactive highlighting and annotating. Since individual text is not selectable as a DOM element, each word is wrapped into an individual DOM element that is selectable, hoverable, and has DOM properties that we can assign it. For example, we have the following text:

The cow jumped over the moon.

Which is manipulated to create individual DOM elements by word:

<span>The </span>

<span>cow </span>

<span>jumped </span>

<span>over </span>

<span>the </span>

<span>moon.</span>

And an id is assigned to each element:

<span id="e1">The </span>

<span id="e2">cow </span>

<span id="e3">jumped </span>

<span id="e4">over </span>

<span id="e5">the </span>

<span id="e6">moon.</span>

This markup is the foundation of digitizing highlighting behavior: it allows the starting and ending boundaries of a highlighted section to be selected. It also allows additional highlighting boundary elements to be created before and after highlighted text, as well as provide the ability to interactively toggle highlighting. Without this, we can’t easily parse or identify starting and ending points other than trying to determine it by other methods such as substring-ing text or using positioning details to identify the current word.

How do we Highlight?

With our example, I’ll describe the history of highlighting functionality that I’ve been involved in on the project. For this post, let’s say our desired emulated highlighted behavior is the following, where “cow jumped over” is highlighted in pink, and “over the” is highlighted in blue:

The cow jumped over the moon.

And our HTML markup may look like this to indicate the highlighted layers:

<span id="e1">The </span>

<span id="e2" class="pink">cow </span>

<span id="e3" class="pink">jumped </span>

<span id="e4" class="pink blue">over </span>

<span id="e5" class="blue">the </span>

<span id="e6">moon.</span>

Highlighting Iteration 1

One of the challenges with emulating highlighting is that a DOM element can only have one background color. We can’t easily layer pink and blue highlights over a specific word to give a layered highlighted effect. So in our example, the pink and blue highlighted words will show up fine, but no color combinations will show up because a node cannot have multiple background colors. During my first iteration on this functionality, I implemented behavior to track the history of our highlights per textual node. The following steps are an example of a use case that demonstrates the highlight overlap:

Starting from unhighlighted text, first a user highlights pink:

The cow jumped over the moon.

Next, a user highlights blue:

The cow jumped over the moon.

Next, a user unhighlights blue:

The cow jumped over the moon.

Finally, a user unhighlights pink:

The cow jumped over the moon.

Or another simple use case:

Starting from unhighlighted text, first a user highlights pink:

The cow jumped over the moon.

Next, a user highlights blue:

The cow jumped over the moon.

Next, a user unhighlights pink:

The cow jumped over the moon.

Finally, a user unhighlights pink:

The cow jumped over the moon.

Programmatically, this method required that a history of highlights be stored to each text node in the form of an array. When something was highlighted or unhighlighted, the array was manipulated to remove or add highlights and the last highlight applied to the text node. While this method was fairly simply to implement, it did not allow users to visualize the overlapping highlighted sections. This iteration was in place for over a year, but ultimately it did not appropriately demonstrate overlapped highlights.

Highlighting Iteration 2

In the next iteration, I attempted to implement a method that added opaque, absolute and fixed positioned elements underneath the text, similar to how one might see layers in Photoshop:

In the second iteration, additional colored & opaque nodes were created under the text to provide a layered highlighting effect.

Unhighlighted markup looked like this:

<span id="e1">
    <span class="highlights"></span>
    The 
</span>
<span id="e2" class="pink">
  <span class="highlights"></span>
    cow 
</span>
<span id="e3" class="pink">
    <span class="highlights"></span>
    jumped
</span>
<span id="e4" class="pink blue">
    <span class="highlights"></span>
    over
</span>
<span id="e5" class="blue">
    <span class="highlights"></span>
    the
</span>
<span id="e6">
    <span class="highlights"></span>
    moon.
</span>

And highlighted markup looked like this:

<span id="e1">
    <span class="highlights"></span>
    The 
</span>
<span id="e2" class="pink">
    <span class="highlights">
        <span class="highlight_pink"></span>
    </span>
    cow 
</span>
<span id="e3" class="pink">
    <span class="highlights">
        <span class="highlight_pink"></span>
    </span>
    jumped 
</span>
<span id="e4" class="pink blue">
    <span class="highlights">
        <span class="highlight_pink"></span>
        <span class="highlight_blue"></span>
    </span>
    over 
</span>
<span id="e5" class="blue">
    <span class="highlights">
        <span class="highlight_blue"></span>
    </span>
    the 
</span>
<span id="e6">
    <span class="highlights"></span>
    moon.
</span>

In the above markup, the following should be noted:

Each span.highlights node is absolutely positioned with a width and height matching the text node.
Each span.highlights span node (e.g. a node with the highlight_pink class) has a width and height of 100%, a background color defined in CSS, and an opacity that is scaled based on the number of highlights.
Whenever highlights are toggled in the text, the children nodes of span.highlights are manipulated (added or removed), as well as the opacity.

While this functionality provides a nice highlighted layering effect, absolute positioning is probably my least favorite thing to work with in cross browser development, and it did not always behave as expected. Specifically, IE and Chrome behaved somewhat as expected, but Firefox did not. This absolute positioning also caused problems with other absolutely positioned elements on the page.

Additionally, this markup was subject to cause significant performance issues. In one case, content with 44,000 words (and text nodes) alone caused performance implications, but additional highlighting layering caused extreme performance pain, to the point that Chrome would not load the content and the content would take more than 30 seconds to load in Firefox, so I went searching for a better solution.

Highlighting Iteration 3

Finally, in the most recent iteration, after identifying that Iteration 2 produced significant performance issues, after more research, I came across the jQuery xColor plugin. This plugin allows you to do mathematical operations on colors, such as combining colors. While the plugin itself only lets you combine 2 colors at one time, I created a method to combine multiple opaque layers:

$.each($('span.' + highlighted_class), function(i, el) {
    var current = $(el);
    var highlight_colors = current.data('highlight_colors');
    if(highlight_colors) {
      highlight_colors.push(hex);
    } else {
       highlight_colors = new Array(hex);
    }
    var current_hex = '#FFFFFF';
    var opacity = 0.4 / highlight_colors.length;
    $.each(highlight_colors, function(i, color) {
        var new_color = $.xcolor.opacity(current_hex, color, opacity);
        current_hex = new_color.getHex();
     });
     current.css('background', current_hex);
     current.data('highlight_colors', highlight_colors);
});

Step by step, the above code does, for each span node with a specific highlight:

Retrieve the array of highlights applied to that node, or create a new array with the new highlight color.
For each highlight, layer an opaque version of that highlight on top of the summation of colors.
Set the background to the final combination of layered highlights.
Store the new array of highlights applied to that node.

The markup is back to the original markup with no additional children elements per text node. Each highlighting interaction triggers a recalculation of the background color per text node based on the data stored to that node.

Conclusion

Significant limitations during this work have included the described performance limitations, as well as the inability to set multiple background images on a DOM element. Absolutely positioning, while valuable at times, proved to be quite challenging because of other absolute and fixed positioned elements on the page. In addition to emulating this highlighting behavior, there are additional interactive requirements included with the interactive highlighting.

Example of additional features needed in interactive highlighting tool.

Additional UI functional requirements include:

Functionality to show paragraph numbers, and hide all paragraph numbers when they have no visible children (Hint: advanced jQuery selectors are used here).
Ability to toggle display of unhighlighted text. The […] in the above image trigger the unhighlighted text in that section to display, while the left and right arrows trigger the unhighlighted text in that section to be hidden.
Ability to toggle display of highlighted text, similar to the toggle of unhighlighted text.
Ability to toggle between a “read” and “edit” mode for owners of the text, which allows for these users to interactively add additional highlights and dynamically modify the markup. In the edit mode, additional markup is added to identify these highlighted layers.
Ability to toggle display of the annotation. In the above image, clicking on the green asterisk toggles this display. No asterisk is shown if there is no annotation.
Ability for highlights to encompass HTML nodes that are not individual span elements. For example, highlighted sections may encompass multiple paragraphs and headers of <span> nodes, which is why simply adding a wrapping element to highlights will not work.

One might suggest we go to a better tool to manage content markup, but ultimately these types of markup tools do not provide the interactivity we seek, and they require that the end user have knowledge of HTML markup, which is not always the case.

CSS sprites: The easy way?

2013-01-21T00:00:00+00:00

I’ve always been interested in the use of CSS sprites to speed up page load times. I haven’t had a real chance to use them yet but my initial reaction was that sprites would be quite painful to maintain. In my mind, you would have to load up the sprite into Gimp or Photoshop, add the new image and then create the css with the right coordinates to display the image. Being a guy with very little image editing skills, I felt that managing multiple images frequently would be quite time consuming. Recently, I was dealing with some page load times for a client and the use of sprites for the product listing pages came up as an option to speed them up. I knew the client wouldn’t have time to create sprites for this so I went searching for a command line tool that would allow me to create sprites. I was quite happy when I stumbled upon Glue.

Glue is free program that will take a directory of images and create a png sprite and a css file with the associated CSS classes. It has a ton of useful options. A few of the ones I thought were handy was being able to prefix the path to the image with a url instead of a relative path, being able to downgrade the png format to png8 to make the file sizes much smaller and being able to specify the start of the name of the class with something to help with being able to use them on a dynamic page.

The most basic use of the command is as follows:

glue blog output

In this example, blog is the directory where the images live that you want Glue to create a sprite of and output is the directory where it will generate the blog.png and blog.css file.

The output css file looks like this:

/* glue: 0.2.9.1 hash: f9c9d6aa5b */
.sprite-blog-product2,
.sprite-blog-product1{background-image:url('blog.png');background-repeat:no-repeat}
.sprite-blog-product2{background-position:0px 0px;width:159px;height:200px;}
.sprite-blog-product1{background-position:-159px 0px;width:200px;height:168px;}

The naming convention by default is sprite-$input_directory-$filename. You can override this with a few options. The version number and hash are used by Glue to ensure it doesn’t rebuild the sprite if none of the source images have changed. With these settings, I believe this could be a great program to run as a nightly routine to rebuild the sprites. This is the explanation from the documentation:

By default glue store some metadata inside the generated sprites in order to not
rebuild it again if the source images and settings are the same. Glue set two different
keys, glue with the version number the sprite was build and hash, generated using the 
source images data, name and all the relevant sprite settings like padding, margin 
etc...

I’m still tinkering with all the different options and thinking about how to include the use of this program in our work flow for this client. I’ll make sure to write a follow up with more information as I learn more.

If you have a different command line tool which helps manage sprites, don’t hesitate to leave a comment and let me know!

Paper Source: The Road to nginx Full Page Caching in Interchange

2013-01-03T00:00:00+00:00

Background & Motivation

During the recent holiday season, it became apparent that some efforts were needed to improve performance for Paper Source to minimize down-time and server sluggishness. Paper Source runs on Interchange and sells paper and stationery products, craft products, personalized invitations, and some great gifts! They also have over 40 physical stores which in addition to selling products, offer on-site workshops.

Over the holiday season, the website experienced a couple of instances where server load spiked causing extreme sluggishness for customers. Various parts of the site leverage Interchange’s timed-build tag, which creates static caches of parts of a page (equivalent to Rails’ and Django’s fragment caching). However, in all cases, Interchange is still being hit for the page request and often the pages perform repeated logic and database hits that opens an opportunity for optimization.

The Plan

The long-term plan for Paper Source is to move towards full page nginx caching, which will yield speedily served pages that do not require Interchange to be touched. However, there are several code and configuration hurdles that we have to get over first, described below.

Step 1: Identify Commonly Visited Pages

First, it’s important to recognize which pages are visited the most frequently and to tackle optimization on those pages first, essentially profiling the site to determine where we will gain the most from performance optimization. In the case of Paper Source, popular pages include:

Thumbnail page, or the template where multiple products are shown in list format
Product detail page, or the template that serves the basic product page
Swatching page, or the template that serves a special product page with special product options
Personalization detail page, or a template that serves the special product page for personalizeable products (e.g. wedding invitatations, birth announcements, etc.)

Step 2: Remove Dynamic User Elements on pages of interest

The next step in the process is to remove dynamic elements on the page, by having cookies or AJAX render these dynamic elements. Below are a couple of examples of these dynamic elements on two primary page templates.

The thumbnail page contains two dynamic elements: the mini-cart template, which shows how many items are in the user’s cart, and the log in information, which shows “my account” and “log out” links if the user is logged in, and shows a “log in” link if the user is not logged in.

In addition to the mini-cart and logged in elements, the product page contains additional dynamic elements which signify if a user has added an item to their cart, and presentation of the user’s previously viewed items.

In the examples above, the following changes were applied to replace these dynamic elements:

Mini-cart Component: This section utilizes cookies with the jQuery.cookie plugin. A cookie stored in the browser identifies the number of cart items and the cart subtotal. After the DOM loads, the mini-cart is rendered and displayed if the user has a non-empty cart. These cookies are manipulated whenever the cart contents are modified.
Login Component: This section also reads a browser-stored cookie. If the cookie indicates the user is logged in, the navigation elements are updated to reflect that.
Added to cart Component: On the product detail page, this code was invasively modified to allow items to be added to cart via AJAX. The AJAX call results in an update of the cart cookies specified above. The feedback of the AJAX call is presented to the user to indicate that the item has been successfully added.
Previously Viewed Component: Finally, the product page also contains previously viewed items. This is generated via a cookie that stores recent skus visited by the user, and each sku has an associated cookie that includes information such as the image source, link, description, and price. Because a maximum of three previously viewed items is shown, cookies here of older previously viewed items are deleted to minimize cookie build-up.

Step 3: Implement fully timed-build caching pages

During this incremental process to reach the end goal of full nginx caching, the next step is to implement fully timed-build pages, or use Interchange’s caching mechanism to fully cache these pages and reduce repetitive database hits and backend logic. In this step, the entire page is wrapped in a timed-build tag, which results in writing and serving a static cached file for that page. While this step is not a necessity, it does allow for us to deploy and test our changes in preparation for nginx caching. In adddition to giving us an opportunity to work out kinks, this step also gives us an added bump in performance because several of these page templates have no caching at all.

Step 4: Reproduce redirect logic outside of Interchange

Next up, we plan to move logic that handles page redirects outside of Interchange to nginx. At the moment, Interchange is responsible for handling 301 redirects on old product and navigation pages. This will need to be moved to nginx redirects to minimize the hits on Interchange here.

Step 5: Implement nginx architecture on camps

Another non-trivial step in this process will be to implement nginx architecture on DevCamps (or camps). DevCamps is an open source tool developed by End Point for developing on multiple instances of copies of the production server. Camps are heavily used for Paper Source because several End Point and internal Paper Source employees simultaneously work on different projects on their development instances or camps. Nginx caching will need to be set up to also work with the camp system in place.

Step 6: Turn nginx caching on!

Finally, we can turn on nginx caching for specific pages of interest. Nginx will then serve these fully cached pages and will avoid Interchange entirely. Cookies and AJAX will still be used to render the dynamic elements on the fully cached pages. While we’d ideally like to cache every page on the site except for the cart, checkout and my account pages, it makes more sense to find the bottlenecks and tackle them incrementally.

Where are we now?

At the moment, I’ve made progress on steps 1-3 for several subsets of pages, including the thumbnail and product detail pages. I plan to continue these steps for additional bottleneck pages. I have worked out out a couple of minor kinks throughout the recent progress, but things have been progressing well. Richard plans to make progress on the nginx related tasks in preparation for reaching the end goal.

Speeding Up Integration Tests with PostgreSQL

2012-06-22T00:00:00+00:00

Many people tend to say they don’t want to write tests. One of the reasons is usually that the tests are too slow. The tests can be slow because they are written in a bad way. They can also be slow because of slow components. One such component is usually a database.

The great thing about PostgreSQL is that all types of queries are transactional. It simply means that you can start a transaction, then run the test, which can add, delete and update all the data and database structure it wants. At the end of the integration test, there should be called rollback which just reverts all the changes. It means the next test will always have the same database structure and data before running, and you don’t need to manually clear anything.

For running the integration tests we need a test database. One of the most important things when running test is speed. Tests should be fast, programmers don’t like to wait ages just to know that there is something wrong.

We can also have a read only databases for the tests. Then you don’t need to worry about the transactions, however you always need to ensure the tests won’t change anything. Even if you assume your tests won’t make any changes, it is always better to use a new transaction for each test and rollback at the end.

The main idea for fast integration tests using PostgreSQL is that those tests don’t change anything in the database. If they don’t change, we don’t need to worry about some possible data loss when the database suddenly restarts. Then we can just restart the tests. Tests should prepare the data before running, assuming the database is in unknown state.

This database should be as fast as possible, even if it means losing data when some unusual things happen. Normally PostgreSQL works really great when someone turns off the server plug suddenly or kills the database process. It just doesn’t lose the data.

However we really don’t need this stuff when running the tests. The database can be loaded before running tests. If the database is suddenly shut down, we should restart the tests.

The simplest thing is to change a couple of settings which enable great secure writes, however it slows down the database. We don’t need to have secure writes, they are only important when something crashes. Then we should restart all the components used for integration tests and load database before testing.

For testing I will use pgbench program which makes a test similar to TPC-B. The tests prepare the data in four tables and then performs a simple transaction:

BEGIN;

UPDATE pgbench_accounts SET abalance = abalance + :delta WHERE aid = :aid;

SELECT abalance FROM pgbench_accounts WHERE aid = :aid;

UPDATE pgbench_tellers SET tbalance = tbalance + :delta WHERE tid = :tid;

UPDATE pgbench_branches SET bbalance = bbalance + :delta WHERE bid = :bid;

INSERT INTO pgbench_history (tid, bid, aid, delta, mtime) VALUES (:tid, :bid, :aid, :delta, CURRENT_TIMESTAMP);

END;

where the params are randomly chosen during execution.

The database is created normally using SQL query:

  CREATE DATABASE pbench;

Before running the tests, pgbench has to prepare initial data. This is done using the -i param. My computer is not very slow, so the default size of the database is too small, I used a quite larger database size using the -s 25 param. This way the database size is about 380MB, including indexes.

  ./pgbench -h localhost pbench -i -s 25

The first test will just run using standard PostgreSQL configuration settings and will run for 5 minutes (the -T 300 param).

  ./pgbench -h localhost -T 300 pbench

The initial results show about 80 transactions per second (tps).

  number of clients: 1
  number of threads: 1
  duration: 300 s
  number of transactions actually processed: 23587
  tps = 78.621158 (including connections establishing)
  tps = 78.624383 (excluding connections establishing)

You may probably noticed the “number of clients” and “number of threads” values. It is the scenario where you have sequential tests, so all of them are run one by one. However integration tests written in a good way can be run in parallel, so let’s run the pgbench once again, but with three threads and three clients.

  ./pgbench -h localhost -c 3 -j 3 -T 300 pbench

The results show that it is a little bit faster now:

  number of clients: 3
  number of threads: 3
  duration: 300 s
  number of transactions actually processed: 29782
  tps = 99.268609 (including connections establishing)
  tps = 99.273464 (excluding connections establishing)

Let’s now change the PostgreSQL settings to some more dangerous, so it can lose some data when shut down suddenly, but in fact I don’t care as all the data is loaded just before running the test.

I’ve written at the end of the postgresql.conf file the following lines:

  fsync = off                # turns forced synchronization on or off
  synchronous_commit = off   # synchronization level; on, off, or local
  full_page_writes = off     # recover from partial page writes

Those changes need a database restart, and after restarting PostgreSQL, I just run the pgbench tests once again.

All the results are in the following table:

<table id="testres">
    <thead>
      <tr>
        <th class="noborder"> </th>
        <th colspan="3"> number of clients and threads </th>
      </tr>
      <tr>
        <th class="noborder"> </th>
        <th>1
        </th><th>2
        </th><th>3
      </th></tr>
    </thead>
    <tbody>
      <tr>
        <td class="name">normal settings</td>
        <td>78 tps</td>
        <td>80 tps</td>
        <td>99 tps</td>
      </tr>
      <tr>
        <td class="name">dangerous settings</td>
        <td>414 tps</td>
        <td>905 tps</td>
        <td>1215 tps</td>
      </tr>
      <tr>
        <td class="name">change ratio</td>
        <td>531 %</td>
        <td>1131 %</td>
        <td>1227 %</td>
      </tr>
    </tbody>
</table>

As you can see, you can do three simple things to speed up your integration tests using PostgreSQL:

Change default PostgreSQL settings to speed the database up.
Change your tests to run in parallel.
Run each test in one transaction.

This way I managed to speed up the tests from 78 tps to 1215 tps. It means that the integration tests which normally run in 60 minutes, should now run in 4 minutes.

I’ve also played with many other settings which could have some impact on the PostgreSQL speed. They really have, however the impact is so small that I don’t think it is worth mentioning here. Changing those three settings can make the PostgreSQL fast enough.

Interchange Caching Implementation Under Fire

2012-06-15T00:00:00+00:00

Richard and David presented a recent case study on an e-commerce hosting client.

Several Interchange catalogs drive their individually branded storefronts, on top of a standard single-server LAMP stack boosted by an SSD drive.

Last year the sites came under an intense Distributed Denial of Service attack which lasted nearly two weeks. End Point responded immediately and soon engaged third-party DDoS mitigation firms. This experience later prompted an Interchange caching implementation.

Cache population and expiration is difficult for any dynamic web application using sessions, and doubly so for e-commerce sites. Every shopping cart needs a session, but delaying session creation until the first POST submission enables efficient caching for most of the sitemap. Other Interchange caching improvements made it back into the upstream code.

Website Performance Boot Camp at UTOSC 2012

2012-05-21T00:00:00+00:00

I’ll keep brief my last post about this year’s Utah Open Source Conference.

I was asked to give on both day one and day two a talk called “Website Performance Boot Camp” which carried this brief description:

What’s the difference between a snappy website and a sloth that you turn away from in frustration? A lot of little things, usually. It’s rarely worth doing 100% of the optimization you could do, but getting 75% of the way isn’t hard if you know where to look.

We’ll look at HTTP caching, compression, proxying, CDNs, CSS sprites, minification, and more, how to troubleshoot, and what’s best to leave alone when you have limited time or tolerance for risk.

Here is the video recording of the first time I presented the talk. (The technician noted its audio was “a little hot”.)

Use this Website Performance Boot Camp direct YouTube video link if the embedded video doesn’t work for you.

The slides for this Website Performance Boot Camp presentation are available.

Thanks again to the conference organizers and the other speakers and sponsors, and the nice venue Utah Valley University, for making it a great conference!

Profile Ruby with ruby-prof and KCachegrind

2012-05-01T00:00:00+00:00

This week I was asked to isolate some serious performance problems in a Rails application. I went down quite a few paths to determine how to best isolate the issue. In this post I want to document what tools worked most quickly to help find offending code.

Benchmarks

Before any work begins finding how to speed things up, we need to set a performance baseline so we can know if we are improving things, and by how much. This is done with Ruby’s Benchmark class and some of Rail’s Benchmark class.

The Rails guides would have you setup performance tests, but I found this cumbersome on this Rails 2.3.5 application I was dealing with. Initial attempts to set it up were unfruitful, taking time away from the task at hand. In my case, the process of setting up the test environment to reflect the production environment was prohibitively expensive, but if you can automate the benchmarks, do it. If not, use the logs to measure your performance, and keep track in a spreadsheet. Regardless of benchmarking manually or automatically, you’ll want to keep some kind of log of the results keeping notes about what changed in each iteration.

Isolating the Problem

As always, start with your logs. In Rails, you get some basic performance information for free. Profiling code slows down runtime a lot. By reviewing the logs you can hopefully make a first cut at what needs to be profiled, reducing already long profile runs. For example, instead of having to profile an entire controller method, by reading the logs you might notice that it’s just a particular partial which is rendering slowly.

Taking a baseline benchmark

Once you’ve got a sense of where the pain is, it’s easy to get a benchmark for that slow code as a baseline.

module SlowModule
  def slow_method
    benchmark "SlowModule#slow_method" do
      #my slow code
    end
  end
end

Look to your log files to see results. If for some reason, you’re outside your Rails enviornment, you can use Ruby’s Benchmark class directly.

require 'benchmark'
result = Benchmark.ms do
  #slow code
end
puts result

This will tell you the process time in milliseconds and give you a precise measurement to compare against.

Profiling with ruby-prof

First, setup ruby-prof. Once installed, you can add these kinds of blocks around your code.

require 'ruby-prof'

module SlowModule
  def slow_method
    benchmark "SlowModule#slow_method" do
      RubyProf.start
      # your slow code here
      results = RubyProf.end
    end
    File.open "#{RAILS_ROOT}/tmp/SlowModule#slow_method_#{Time.now}", 'w' do |file|
      RubyProf::CallTreePrinter.new(results).print(file)
    end
  end
end

Keep in mind that profiling code will really slow things down. Make sure to collect your baseline both with profiling and without, to make sure you’re doing apples-to-apples comparison.

By default ruby-prof measures process time, which is the time used by a process between any two moments. It is unaffected by other processes concurrently running on the system. You can review the ruby-prof README for other types of measurements including memory usage, object allocations and garbage collection time.

If you choose to measure any of these options, make sure your Ruby installation has the tools a profiler needs to collect data. Please see the Rails guides for guidance on compiling and patching Ruby.

Interpreting the Data with KCachegrind

At this point you should have a general sense of what code is slow having reviewed the logs. You’ve got a benchmark log setup with baseline measurements to compare to. If you’re going to Benchmark while your profiling, make sure your baseline includes the profile code; it will be much slower! Remember we want an apples-to-apples comparison! You’re ready to start profiling and identifying the root source of the performance problems.

After manually or automatically running your troubled code with the profiling block above, you can open up the output from ruby-prof and quickly find it not to be human friendly. Fortunately, KCachegrind turns that mess into something very useful. I found that my Ubuntu installation had a package for it already built, so installation was a breeze. Hopefully things are as easy for you. Next simply open your result files and start reviewing there results.

The image above shows what’s called a “call graph” with the percentages representing the relative amount of time that method uses for the duration of the profile run. The CacheableTree#children method calls Array#collect and takes up more then 90% of the runtime. The subsequent child calls are relatively modest in proportion. It’s clear we can’t modify Array#collect so let’s look at CacheableTree#children.

module CacheableTree
  def children(element = @root_element)
    full_set.collect { |node| if (node.parent_id == element.id)
      node
    }.compact
  end
end

Defined elsewhere, full_set is an array of Ruby objects. This is common performance optimization in Rails; collecting data looping through arrays works well with a small data set, but quickly becomes painful with a large one. It turned out in this case that full_set had 4200+ elements. Worse yet the children method was being called recusrively on each of them. Yikes!

At this point I had to decide how to optimize. I could go for broke and completely break the API and try and clean up the mess, or I could see if I could collect the data more quickly, some other way. I looked at how the full_set was defined and found I could modify that query to return a subset of elements rather easily.

module CacheableTree
  def children(element = @root_element)
    FormElement.find_by_sql(...) #the details aren't important
  end
end

By collecting the data directly via a SQL call, I was able to cut my benchmark by about 20%. Not bad for a single line change! Let’s see what the next profile told us.

The above is another view of the profile KCachegrind provides. It’s essentially the same information, but in table format. There were a few indicators that my optimization was helpful:

The total process_time cost had dropped
The amount of time spent in each function seemed to better distributed—I didn’t have a single method soaking up all the process time
Most of the time was spent in code wasn’t mine!

Although, we still saw 66% of time spent in the children method, we could also see that 61% of the time was spent in ActiveRecord::Base. Effectively, I had pushed the ‘slowness’ down the stack, which tends to mean better performance. Of course, there were LOTS of database calls being made. Perhaps some caching could help reduce the number of calls being made.

module CacheableTree
  def children(element = @root_element)
    @children ||= {}
    @children[element] ||= FormElement.find_by_sql(...) #the details aren't important
  end
end

This is called memoization and let’s us reuse this expensive method’s results within the page load. This method took another 10% off the clock against the baseline. Yay!

Knowing When to Stop

Performance optimization can be really fun, especially once all the infrastructure is in place. However, unless you have unlimited budget and time, you have to know when to stop. For a few lines of code changed, the client would see ~30% performance improvement. It was up to them to decide how much further to take it.

If allowed, my next step would be to make use of the applications existing dependence on redis, and add the Redis-Cacheable gem. It allows you to marshal Ruby objects in and out of a redis server. The application already makes extensive use of caching, and this page was no exception, but when the user modified the page in a way that expired the cache, we would hit this expensive method again, unnecessarily. Based on the call graph above, we could eliminate another ~66% of the call time, and perhaps, by pre-warming this cache, could help the user to minimize the chances of experiencing the pain of slow browsing!

RailsConf 2012: Day Two

2012-04-24T00:00:00+00:00

I’m here in Austin for Day 2 of RailsConf 2012. Here’s a quick run-down of a few talks I attended today. Read about a couple of sessions I attended on Day 1 here.

Let’s Make the Web Faster

One of the talks I attended on Day 2 was Let’s Make the Web Faster — tips from the trenches @ Google by Ilya Grigorik. Ilya works on the MTWF (make the web faster) team at Google; one of their goals is not only to make Google’s web applications faster, but also to make a bigger impact on web performance by developing and contributing to tools. I found a few main takeaways from Ilya’s talk:

Navigation Timing Spec: The navigation timing spec defining an interface for web applications to access timing information related to navigation and elements. You can look at this timing information by checking out the performance object in a console:

Example output of navigation timing spec data.
Site Speed Tools in Google Analytics: Google Analytics now has elements of the page speed broken down that allows you to compare page load times and examine what the bottlenecks in the system are. To get there, you go to Content => Site Speed => Page Timings in Google Analytics. You can also measure and examine this data using Segments and Advanced segments.

Example metrics shown on Google Site Speed breakdown.
WebPageTest.org is a tool that we’ve blogged about before a few times. Now it also provides a video of how the page loads to examine how the page load is perceived.
Speed Index: Speed index is a new metric for measuring “visual completeness” of a page loading. This is an interesting/valuable metric to look at when you want to try to understand and improve user perceived latency. For example, can you force loading some images before peripheral scripts to improve the user perceived latency?
A few Google tools that Ilya reviewed were PageSpeed Online and its API, PageSpeed SDK, and PageSpeed Service, which is in beta mode.
Ilya’s recap was that you should a) measure performance b) optimize performance (with specific attention to improve user perceived latency) and c) use tools to automate performance.

Check out the full presentation here.

Presenters and Decorators: A Code Tour

Another talk I attended was Presenters and Decorators: A Code Tour by Mike Moore. He started with a brief explanation of the justification (more later) for presenters — as a “pain driven developer”, Mike tries to reduce the pain of advanced view logic with presenters. Mike then went through a complex example of rendering which mapped through a view to a helper to an app/component file to a class in the library back to the app to a haml view; this is painful to dissect, maintain, and test, so presenters are a good option here to avoid this spaghetti code.

Mike then went through coding examples and thought processes that lead to using a presenter-like solution. He presented an example where a view contained view-specific logic and instantiation and use of variables. The natural first step for refactoring the view here would be to pull the logic into the model, but the argument against this is that because it’s view specific (as opposed to domain specific), it should not live in the model. Another progression step in pulling the logic out of the view would be to create a class that’s initialized in the view, and the various instance methods on that class are called in the view. Here Mike introduced the ActiveDecorator plugin. This plugin automagicallly includes class decorators for instances in the view, which eliminates the need for instantiation of the presenter object. Mike also reviewed a standard serialization example, where a JSON-ified view can have logic that should probably live elsewhere. One option here is to create a class serializer option which has methods to render data to json. Another option here is to use ActiveModel::Serializers which looks for the rendering of JSON to remove logic from the JSON rendering view. These examples and another thorough code example are best viewed in the slides.

Mike explained that it’s difficult to define Presenters and that he perceives presenters on a spectrum between the model and view which is an object that represents state and behavior of the view. The justification for presenters is to write easier to read, easier to maintain, easier to hand off to a designer, and easier to test code. Mike also noted that this isn’t something that a Rails noob should get into and also that if it’s a problem that you don’t see in your code base, it’s not a problem you necessarily need to solve.

Ten 42 Things You Didn’t Know Rails Could Do

Another fun talk I attended on Day 2 was Ten Things You Didn’t Know Rails Could Do by James Edward Gray II. In list-presentation style, James reviewed 42 things you can do with Rails that you might not know about [which work in Rails 3.2.3 and may or may not work in previous versions of Rails]. This talk reminded me of some of the protips I picked up from taking the Rails Best Practices course over at Code School because there were so many usable tips. I summarized the presentation into my own list of takeaways, but note that I left out some of the advanced tips that required extensive code examples. View the slides here.

The command rake notes will output all your FIXME, TODO, and OPTIMIZE notes in your Rails application. You can also pass in custom annotations to output here as well.
In the console, you can call helper methods (e.g. helper.number_to_currency(100)
You can automagically add indexes via migration shorthand (e.g. rails g resource user name:index). And you can automagically add associations (e.g. rails g resource article user:references or rails g resource article:belongs_to) via migration shorthand.
rake db:migrate:status will tell you where your migration is at
You can use the pluck command instead of the map command. (e.g. User.pluck(:email), User.uniq.pluck(:email))
You can override association methods to “hook” in custom behavior.
You can use limitless strings in PostgreSQL. This requires a bit of code shared in the presentation. I know, my fellow PostgreSQL expert coworkers are going to balk that this is on the list, but unfortunately Rails doesn’t allow this without some code.
You can utilize PostgreSQL’s full text search with a bit of code.
You can use a different database for each user. This one could be great for multi-domain ecommerce, which I recently blogged about.
You can merge nested hashes via the deep_merge method.
You can remove a specific key from a hash via the except method.
You can add defaults to a hash without overriding via the reverse_merge method.
You can override form helpers, and a lot of other view stuff, but this was a little hard to encapsulate in a blog post, so it’ll be best viewed in the slides.
You can route exceptions (e.g. match “/404”, to “errors#not_found”)
You can route to Sinatra. The example James gave here was to allow display of the Resque web interface inside a Rails application, but we have another client who would benefit from this technique as well.

Ta Da

Hopefully this article was more Rails-ey than yesterday’s RailsConf 2012: Day One post which focused on Backbone.js and CoffeeScript. I’m looking forward to a couple of tomorrow’s sessions—and hopefully they will be blogworthy. Stay tuned!