Updated NoSQL benchmark: Cassandra, MongoDB, HBase, Couchbase

2015-06-04T00:00:00+00:00

Back in April, we published a benchmark report on a number of NoSQL databases including Cassandra MongoDB, HBase, and Couchbase. We endeavored to keep things fair and configured as identically as possible between the database engines. But a short while later, DataStax caught two incorrect configuration items, in Cassandra and HBase, and contacted us to verify the problem. Even with the effort we put in to keeping everything even, a couple erroneous parameters slipped through the cracks! I’ll save the interesting technical details for another post coming soon, but once that was confirmed we jumped back in and started work on getting corrected results.

With the configuration fixed we re-ran a full suite of tests for both Cassandra and HBase. The updated results have published a revised report that you can download in PDF format from the DataStax website (or see the overview link).

The revised results still show Cassandra leading MongoDB, HBase, and Couchbase in the various YCSB tests.

For clarity the paper also includes a few additional configuration details that weren’t in the original report. We regret any confusion caused by the prior report, and worked as quickly as possible to correct the data. Feel free to get in contact if you have any questions.

New NoSQL benchmark: Cassandra, MongoDB, HBase, Couchbase

2015-04-13T00:00:00+00:00

Today we are pleased to announce the results of a new NoSQL benchmark we did to compare scale-out performance of Apache Cassandra, MongoDB, Apache HBase, and Couchbase. This represents work done over 8 months by Josh Williams, and was commissioned by DataStax as an update to a similar 3-way NoSQL benchmark we did two years ago.

The database versions we used were Cassandra 2.1.0, Couchbase 3.0, MongoDB 3.0 (with the Wired Tiger storage engine), and HBase 0.98. We used YCSB (the Yahoo! Cloud Serving Benchmark) to generate the client traffic and measure throughput and latency as we scaled each database server cluster from 1 to 32 nodes. We ran a variety of benchmark tests that included load, insert heavy, read intensive, analytic, and other typical transactional workloads.

We avoided using small datasets that fit in RAM, and included single-node deployments only for the sake of comparison, since those scenarios do not exercise the scalability features expected from NoSQL databases. We performed the benchmark on Amazon Web Services (AWS) EC2 instances, with each test being performed three separate times on three different days to avoid unreproduceably anomalies. We used new EC2 instances for each test run to further reduce the impact of any “lame instance” or “noisy neighbor” effect on any one test.

Which database won? It was pretty overwhelmingly Cassandra. One graph serves well as an example. This is the throughput comparison in the Balanced Read/Write Mix:

Our full report, Benchmarking Top NoSQL Databases, contains full details about the configurations, and provides this and other graphs of performance at various node counts. It also provides everything needed for others to perform the same tests and verify in their own environments. But beware: Your AWS bill will grow pretty quickly when testing large numbers of server nodes using EC2 i2.xlarge instances as we did!

Earlier this morning we also sent out a press release to announce our results and the availability of the report.

Update: See our note about updated test runs and revised report as of June 4, 2015.

Riding the Elasticsearch River on a CouchDB: Part 1

2015-01-12T00:00:00+00:00

As you may guessed from my perfect tan and rugged good looks, I am Phunk, your river guide. In this multi-part series, I will guide us through an exploration of Elasticsearch, its CouchDB/BigCouch River plugin, its source, the CouchDB document store, and the surrounding flora and fauna that are the Ruby on Rails based tools I created to help the DPLA project manage this ecosystem.

Before we get our feet wet, let’s go through a quick safety briefing to discuss the terms I’ll be using as your guide on this trip. Elasticsearch: A schema-less, JSON-based, distributed RESTful search engine. The River: An Elasticsearch plugin that automatically indexes changes in your upstream (heh) document store, in real-time. CouchDB: The fault-tolerant, distributed NoSQL database / document store. DPLA: The Digital Public Library of America open source project for which all this work was done.

Let’s put on our flotation devices, don our metaphor helmets and cast off.

In an Elasticsearch + River + CouchDB architecture, all things flow from the CouchDB. For the DPLA project, we wanted to manage (create, update and delete) documents in our CouchDB document repository and have those changes automagically reflected in our Elasticsearch index. Luckily, CouchDB publishes a real-time stream (heh) of updates to its documents via its cleverly named “_changes” feed. Each change in that feed is published as a stand-alone JSON document. We’ll look at that feed in more detail in a bit.

The River bridges (heh) the gap between CouchDB’s _changes feed and Elasticsearch index. The plugin runs inside Elasticsearch, and makes a persistent TCP connection to CouchDB’s _changes endpoint. When a new change is published to that endpoint, the River passes the relevant portions of that JSON up to Elasticsearch, which then makes the appropriate change to its index. Let’s look at a simple timeline of what the River would see from the _changes feed during the creation of a new document in CouchDB, and then an update to that document:

A document is created in CouchDB, the _changes feed emits:

{
  "seq":1,
  "id":"test1",
  "changes":[{"rev":"1-967a00dff5e02add41819138abb3284d"}],
  "doc":{
    "_id":"test1",
    "_rev":"1-967a00dff5e02add41819138abb3284d",
    "my_field":"value1"
  }
}

That same document is updated in CouchDB, the _changes feed emits:

{
  "seq":2,
  "id":"test1",
  "changes":[{"rev":"2-80647a2a9498f5c124b1b3cc1d6c6360"}],
  "doc":{
    "_id":"test1",
    "_rev":"2-80647a2a9498f5c124b1b3cc1d6c6360",
    "my_field":"value2"
  }
}

It’s tough to tell from this contrived example document, but the _changes feed actually includes the entire source document JSON for creates and updates. (I’ll talk more about that in part 2.) From the above JSON examples, the River would pass the inner-most document containing the _id, _rev and my_field data up to Elasticsearch. Elasticsearch uses that JSON to update the corresponding document (keyed by _id) in its search index and voila, the document you updated in CouchDB is now updated in your Elasticsearch search index in real-time.

We have now gotten our feet wet with how a document flows from one end to the other in this architecture. In part 2, we’ll dive deeper into the DevOps-heavy care, feeding, monitoring and testing of the River. We’ll also look at some slick River tricks that can transform your documents before Elasticsearch gets them, and any other silly River puns I can come up with. I’ll also be reading the entire thing in my best David Attenborough impression and posting it on SoundCloud.

CouchDB pagination with couchdb-python

2013-07-02T00:00:00+00:00

I’ve been working with couchdb-python v0.8 to meet some of a project’s CouchDB needs and ran into an unfortunate shortcoming.

Retrieving database rows is as easy as:

for row in db.view(mapping_function):
    print row.key

However, all the rows will be loaded into memory. Now, for small databases this is not a problem but lucky me is dealing with a 2.5 million+ document database. Long story short: kaboom!

There is an excellent blog post by Marcus Brinkmann that details the memory issue and also provides a pagination solution. However, couchdb-python v0.9 (released 2013-04-25) does provide its own solution: the iterview method. If you’ve been able to manage small databases with v0.8 but are anticipating larger and larger databases, be sure to upgrade!

PostgreSQL as NoSQL with Data Validation

2013-06-03T00:00:00+00:00

PostgreSQL is a relational database with many great features. There are also many so called NoSQL databases, some of them, like CouchDB, are document databases. However the document in CouchDB is automatically enhanced with a “_id” field, if it is not present. When you want to get this one document, you can use this “_id” field—it behaves exactly like the primary key from relational databases. PostgreSQL stores data in tables’ rows while CouchDB stores data as JSON documents. On one hand CouchDB seems like a great solution, as you can have all the different data from different PostgreSQL tables in just one JSON document. This flexibility comes with a cost of no constraints on the data structure, which can be really appealing at the first moment and really frustrating when you have a huge database and some of the documents contain bad values or there are missing some fields.

PostgreSQL 9.3 comes with great features which can turn it into a NoSQL database, with full transaction support, storing JSON documents with constraints on the fields data.

Simple Example

I will show how to do it using a very simple example of a table with products. Each product has a name, description, some id number, price, currency and number of products we have in stock.

PostgreSQL Version

The simple table in PostgreSQL can look like:

CREATE TABLE products (
    id SERIAL PRIMARY KEY,
    name TEXT,
    description TEXT,
    price DECIMAL(10,2),
    currency TEXT,
    in_stock INTEGER
);

This table allows us to insert products like:

INSERT INTO products (name, description, price, currency, in_stock) VALUES ('shoes', 'blue shoes', 12.34, 'dollars', 5);

Unfortunately the above table also allows for adding rows missing some important information:

INSERT INTO products (name, description, price, currency, in_stock) VALUES ('', null, -20, 'handa', -42);

This should be fixed by adding constraints in the database. Assume that we want to always have unique not empty name, not empty description, non negative price and in_stock, and the currency should always be dollars. The table with such constraints is:

CREATE TABLE products (
    id SERIAL PRIMARY KEY,
    name TEXT UNIQUE NOT NULL,
    description TEXT NOT NULL,
    price DECIMAL(10,2) NOT NULL,
    currency TEXT NOT NULL,
    in_stock INTEGER NOT NULL,
    CHECK (length(name) > 0),
    CHECK (description IS NOT NULL AND length(description) > 0),
    CHECK (price >= 0.0),
    CHECK (currency = 'dollars'),
    CHECK (in_stock >= 0)
); >

Now all the operations, like adding or modifying a row, which violate any of those constraints, just fail. Let’s check:

postgres=# INSERT INTO products (name, description, price, currency, in_stock) VALUES ('shoes', 'blue shoes', 12.34, 'dollars', 5);
INSERT 0 1
postgres=# INSERT INTO products (name, description, price, currency, in_stock) VALUES ('shoes', 'blue shoes', 12.34, 'dollars', -1);
ERROR:  new row for relation "products" violates check constraint "products_in_stock_check"
DETAIL:  Failing row contains (2, shoes, blue shoes, 12.34, dollars, -1).

NoSQL Version

In CouchDB the inserted row in the above table, would be just a JSON looking like this:

{
    "id": 1,
    "name": "shoes",
    "description": "blue_shoes",
    "price": 12.34,
    "currency": "dollars",
    "in_stock": 5
}

The Trivial Solution

In PostgreSQL we can store this JSON as a row in the products table:

CREATE TABLE products (
    data TEXT
);

This works like most of the NoSQL datatabases, no checks, no errors with bad fields. As a result, you can modify the data the way you want, the problem begins when your application expects that the price is a number, and you get a string there, or there is no price at all.

Validate JSON

CouchDB validates JSON before saving the document into database. In PostgreSQL 9.2 there is the nice type for that, it is named JSON. The JSON type can store only a proper JSON, there is validation performed before converting into this type.

Let’s change the definition of the table to:

CREATE TABLE products (
    data JSON
);

We can insert correct JSON into this table:

postgres=# INSERT INTO products(data) VALUES('{
    "id": 1,
    "name": "shoes",
    "description": "blue_shoes",
    "price": 12.34,
    "currency": "dollars",
    "in_stock": 5
}');
INSERT 0 1
postgres=# SELECT * FROM products;
               data
----------------------------------
 {                               +
     "id": 1,                    +
     "name": "shoes",            +
     "description": "blue_shoes",+
     "price": 12.34,             +
     "currency": "dollars",      +
     "in_stock": 5               +
 }
(1 row)

This works, but inserting not a valid JSON ends with an error:

postgres=# INSERT INTO products(data) VALUES('{
    "id": 1,
    "name": "shoes",
    "description": "blue_shoes",
    "price": 12.34,
    "currency": "dollars",
    "in_stock": 5,
}');
ERROR:  invalid input syntax for type json
LINE 1: INSERT INTO products(data) VALUES('{
                                          ^
DETAIL:  Expected string, but found "}".
CONTEXT:  JSON data, line 5: ...,
    "currency": "dollars",
    "in_stock": 5,
}

The problem with formatting can be hard to notice (I’ve added comma after the last field, JSON doesn’t like it).

Validating Fields

OK, so we have a solution which looks almost like the first native PostgreSQL solution: we have data which validates. It doesn’t mean the data is sensible.

Let’s add checks for validating the data.

In PostgreSQL 9.3, which has not been released yet, there are some new great features for manipulating JSON values. There are defined operators for the JSON type, which give you easy access to the fields and values.

I will use only one operator “-»”, but you can find more information in PostgreSQL documentation.

I also need to validate the types of the fields, including id field. This is something Postgres just checks because of the types definitions. I am going to use some other syntax for the checks, as I want to name it. It will be easier to look at problem with specific field instead of searching through the whole huge JSON.

The table with the constraints looks like this:

CREATE TABLE products (
    data JSON,
    CONSTRAINT validate_id CHECK ((data->>'id')::integer >= 1 AND (data->>'id') IS NOT NULL ),
    CONSTRAINT validate_name CHECK (length(data->>'name') > 0 AND (data->>'name') IS NOT NULL ),
    CONSTRAINT validate_description CHECK (length(data->>'description') > 0  AND (data->>'description') IS NOT NULL ),
    CONSTRAINT validate_price CHECK ((data->>'price')::decimal >= 0.0 AND (data->>'price') IS NOT NULL),
    CONSTRAINT validate_currency CHECK (data->>'currency' = 'dollars' AND (data->>'currency') IS NOT NULL),
    CONSTRAINT validate_in_stock CHECK ((data->>'in_stock')::integer >= 0 AND (data->>'in_stock') IS NOT NULL )
}

The “-»” operator allows me to get the value of a specific field from JSON, check if it exists and validate it.

Let’s add a JSON without a description:

postgres=# INSERT INTO products(data) VALUES('{
    "id": 1,
    "name": "d",
    "price": 1.0,
    "currency": "dollars",
    "in_stock": 5
}');

ERROR:  new row for relation "products" violates check constraint "validate_description"
DETAIL:  Failing row contains ({
    "id": 1,
    "name": "d",
    "price": 1.0,
    "currency...). >

There is one more validation left. The id and name fields should be unique. This can be easily done with two indexes:

CREATE UNIQUE INDEX ui_products_id ON products((data->>'id'));
CREATE UNIQUE INDEX ui_products_name ON products((data->>'name'));

Now when you try to add a JSON document which id which already exists in database, then you will have an error like:

ERROR:  duplicate key value violates unique constraint "ui_products_id"
DETAIL:  Key ((data ->> 'id'::text))=(1) already exists.
ERROR:  current transaction is aborted, commands ignored until end of transaction block

Id Generation

In NoSQL databases the id field is usually some UUID. This is an identifier generated with algorithms with a very small chance of generating the same value, even when you generate them on different machines. So I’m not going to touch it here.

Searching

You can search the JSON data normally like you were searching columns in a table. Let’s search for the most expensive product we have in stock:

SELECT * FROM products WHERE in_stock > 0 ORDER BY price DESC LIMIT 1;

The JSON version is very similar:

SELECT * FROM products WHERE (data->>'in_stock')::integer > 0 ORDER BY (data->>'price')::decimal DESC LIMIT 1;

This query can be very inefficient. It needs to read all the rows, parse JSON fields and check the in_stock and price fields, convert into proper types and then sort. The plan of such a query, after filling the table with 100k rows, looks like this:

                                                        QUERY PLAN
-----------------------------------------------------------------------------------------------------------------------------
 Limit  (cost=9256.48..9256.48 rows=1 width=32) (actual time=412.911..412.912 rows=1 loops=1)
   ->  Sort  (cost=9256.48..9499.05 rows=97027 width=32) (actual time=412.910..412.910 rows=1 loops=1)
         Sort Key: (((data ->> 'price'::text))::numeric)
         Sort Method: top-N heapsort  Memory: 25kB
         ->  Seq Scan on products  (cost=0.00..8771.34 rows=97027 width=32) (actual time=0.022..375.624 rows=100000 loops=1)
               Filter: (((data ->> 'in_stock'::text))::integer > 0)
 Total runtime: 412.939 ms
(7 rows)

The “Seq Scan” line means that PostgreSQL needs to read the whole table. The time of 412 ms is not that bad, but can we make it better?

Fortunately PostgreSQL has a great feature: indexes on expressions, also named as functional indexes. It can store in the index sorted values of some expressions, and if the same expressions occur in a query, then the index can be used.

The indexes I need are:

CREATE INDEX i_products_in_stock ON products(( (data->>'in_stock')::integer ));
CREATE INDEX i_products_price ON products(( (data->>'price')::decimal ));

Notice the double parenthesis, they are required because of the non trivial expression.

The plan now looks a little bit different, after creating indexes and running analyze on the products table:

                                                                    QUERY PLAN
----------------------------------------------------------------------------------------------------------------------------------------------------
 Limit  (cost=0.42..0.55 rows=1 width=32) (actual time=0.041..0.041 rows=1 loops=1)
   ->  Index Scan Backward using i_products_price on products  (cost=0.42..13690.06 rows=100000 width=32) (actual time=0.041..0.041 rows=1 loops=1)
         Filter: (((data ->> 'in_stock'::text))::integer > 0)
 Total runtime: 0.062 ms
(4 rows)

So it is 664k percent faster.

The JSON Advantage

The JSON solution has got one nice feature which the native PostgreSQL hasn’t. The application can add its own fields on the fly without altering any table. JSON field is just a text, however with some validation. The new field won’t be checked by the indexes and constraints I’ve shown you above.

What’s more, you can add a constraint for this field later. This way you can have the best from both worlds: easy data model changing and consistent JSON structure across the database.

On the other hand you could of course add a trigger checking the JSON, before saving it to database, to check the list of available fields. This way you could prevent adding new fields by the application.

Summary

So, I’ve shown you how you can use PostgreSQL as a simple NoSQL database storing JSON blobs of text. The great advantage over the simple NoSQL databases storing blobs is that you can constrain the blobs, so they are always correct and you shouldn’t have any problems with parsing and getting them from the database.

You can also query the database very easily, with huge speed. The ad-hoc queries are really simple, much simpler than the map-reduce queries which are needed in many NoSQL databases.

Quick Thoughts on NoSQL Live Boston Conference

2010-03-11T00:00:00+00:00

I’m back home now from the Boston “NoSQL Live” conference organized by 10gen.com (the MongoDB folks). It was a good event. A lot of stuff covered, a broad range of topics. I have a fair amount to say, but need to digest, review notes, etc. In any case, thanks to 10gen and the various sponsors that made it happen.

Some quick, random thoughts:

Picking a good table at lunch is key: we ended up sitting with four different presenters, including Jonathan Ellis for Cassandra and Alex Feinberg for Voldemort, which happen to be two of the systems I’m personally most interested in using at the moment.
There is an undeniable latest-thing-fan(boy|girl)ism aura surrounding the “NoSQL” brand/meme/whatever, but the various presenters and leading lights in various projects appear to be reasonable and fact-based; don’t let the breathless silliness of fans fool you.
I went in feeling convinced of the desirability of non-relational datastores for specific modeling situations (graphs) and for scalability/availability/volume concerns (Dynamo and BigTable derivatives), while feeling relatively skeptical of “document datastores”. I left feeling basically the same way, though decidedly less skeptical of CouchDB than I previously was.
There is a lot of good thinking and discussion going on in the space, it’s moving very fast, and the future looks bright.

More later. Try to contain your anticipation.