Slony, sl_status and diagnosing a particular type of lag
By Selena Deckelmann
July 31, 2009
During some routine checking on a slony cluster, Greg noticed something curious. Replication was still happening between the master and a couple slaves, but we were seeing our indicator for lag inside of slony increasing.
To check out the status of slony replication, you will typically take a look at the view ‘sl_status’:
mydatabase=# select * from sl_status;
st_origin | st_received | st_last_event | st_last_event_ts | st_last_received | st_last_received_ts | st_last_received_event_ts | st_lag_num_events | st
_lag_time
-----------+-------------+---------------+----------------------------+------------------+----------------------------+---------------------------+-------------------+---------
----------------
2 | 1 | 2697511 | 2008-04-30 02:40:06.034144 | 2565031 | 2008-04-14 15:31:32.897165 | 2008-04-14 16:24:08.81738 | 132480 | 15 days
10:16:03.060499
(1 row)
This view pulls data out of sl_event and sl_confirm, two tables that keep track of the forward progress of replication. Every time there is an event—SYNCs, DDL changes, slony administrative events—a row is added to sl_event. Slony is very chatty …
postgres
Comparing databases with check_postgres
By Greg Sabino Mullane
July 28, 2009
One of the more recent additions to check_postgres, the all-singing, all-dancing Postgres monitoring tool, is the “same_schema” action. This was necessitated by clients who wanted to make sure that their schemas were identical across different servers. The two use cases I’ve seen are servers that are being replicated by Bucardo or Slony, and servers that are doing horizontal sharding (e.g. same schema and database on different servers: which server you go to depends on (for example) your customer id). Oft times a new index fails to make it to one of the slaves, or some function is tweaked on one server by a developer, who then forgets to change it back or propagate it. This program allows a quick and automatable check for such problems.
The idea behind the same_schema check is simple: we walk the schema and check for any differences, then throw a warning if any are found. In this case, we’re using the term “schema” in the classic sense of a description of your database objects. Thus, one of the things we check is that all the schemas (in the classic RDBMS sense of a container of other database objects) are the same, when running the “same_schema” …
database open-source perl postgres
SDCH: Shared Dictionary Compression over HTTP
By Jon Jensen
July 27, 2009
Here’s something new in HTTP land to play with: Shared Dictionary Compression over HTTP (SDCH, apparently pronounced “sandwich”) is a new HTTP 1.1 extension announced by Wei-Hsin Lee of Google last September. Lee explains that with it “a user agent obtains a site-specific dictionary that then allows pages on the site that have many common elements to be transmitted much more quickly.” SDCH is applied before gzip or deflate compression, and Lee notes 40% better compression than gzip alone in their tests. Access to the dictionaries stored in the client is scoped by site and path just as cookies are.
The first client support was in the Google Toolbar for Internet Explorer, but it is now going to be much more widely used because it is supported in the Google Chrome browser for Windows. (It’s still not in the latest Chrome developer build for Linux, or at any rate not enabled by default if the code is there.)
Only Google’s web servers support it to date, as far as I know. Someone intended to start a mod_sdch project for Apache, but there’s no code at all yet and no activity since September 2008.
It is interesting to consider the challenge this will have on HTTP proxies that filter …
browsers hosting networking security compression
Fedora goes up to eleven
By Daniel Browning
July 23, 2009
I upgraded to Fedora Core 11 this week, up from 9. What really surprised me is how fast it is. I don’t recall ever having such a noticably faster and responsive desktop after a distro upgrade. Everything is more responsive and instantaneous. Even though FC9 didn’t seem particularly slow, I realize now that I had been spending time just waiting for the software to catch up. I don’t know where the credit is due. Could be anything from the kernel to XFCE to the apps. But I like it.
Normally, most SELinux issues occur when it is enabled. But after the upgrade I ran into one of the opposite variety: the gedit program threw an error on every file save. The 244605 and 477070 tickets might be the same issue I ran into. I worked around the problem by switching to ‘kate -u’.
I need at least four different text editors in order to enjoy using the computer. How could anyone stand the monotony of a single text editor all day? Right now I’m doing:
- gvim for detailed things
- vanilla vim for when gvim is too awesome.
- kate for vogon poetry
- jedit for when I get that certain IDE feeling (you know the one)
- emacs for inducing carpal tunnel syndrome
It’s also important to switch out text editors at …
redhat linux
OSCON so far! Filesystem information bonanza on Wednesday
By Selena Deckelmann
July 23, 2009
Wednesday was the first official day of OSCON, and I spent it elbow deep in filesystems. The morning was kicked off with Val Aurora delivering a great overview of Btrfs, a new fileystem currently in development. Some of the features include:
- Copy on write filesystem
- Cheap, easy filesystem snapshots
- Dynamically resizable partitions
- Indexed directory structure
- Very simple administration
Val demonstrated basic functionality, including creating snapshots and creating a Btrfs filesystem on top of an ext3 filesystem. Cool stuff! The filesystem is still under heavy development, but seems very promising.
Next I saw Theodore Ts’o, the primary developer behind ext4, talk about the future of filesystems and storage. He referenced a great paper that dives deep into the economics behind SSD (solid state drives) and platter hard drive manufacturing. One interesting calculation was that even if we could convert all the silicon fabs to manufacture flash, would only be able to covert about 12% of the world-wide capacity of hard drive production. Because of this, Theodore believes that it is going to be challenging for the cost of SSDs to drop to the point where it becomes cost competitive with …
conference postgres
Gmail Contacts Notes Converter
By Jon Jensen
July 21, 2009
As I mentioned previously, I recently got a Google Ion phone running Android. I recently began using it as my main mobile phone, and thus needed to finally migrate the contacts from my Nokia 6126 phone to Android.
This is apparently easy to do by first copying all the contacts from the Nokia 6126 internal memory to the SIM card, then moving the SIM card to the Ion and importing the contacts. But that only works if all your contacts fit on the SIM card. If not, they’re truncated, and you have to delete many contacts on the Nokia to fit more, which would be a nonreversable move.
Several posts describe ways to do the export and import, such as this one that didn’t really apply to my phone, and this one that involves VCF export & import which I didn’t see a way to do.
Ultimately I found an article that described Nokia’s PC Suite software that I’d never heard of before, which I downloaded on an old Windows machine and used to download the contacts from the phone via Bluetooth, then export to a CSV file and import into Gmail. So far, so good.
Except as this post and another post describe, then all the contact data showed up in a single Notes field, useless for dialing or emailing.
I …
mobile
pgGearman 0.1 release!
By Selena Deckelmann
July 20, 2009
Yesterday, Brian Aker and Eric Day presented pgGearman: A distributed worker queue for PostgreSQL during the OSCON/SFPUG PgDay.
Gearman is a distributed worker queuing system that allows you to farm work out to a collection of servers, and basically run arbitrary operations. The example they presented was automating and distributing the load of image processing for Livejournal. For example, everyone loves to share pictures of their kittens, but once an image is uploaded, it may need to be scaled or cropped in different ways to display in different contexts. Gearman is a tool you can use to farm these types of jobs out.
So, in anticipation of the talk, I worked with Eric Day on a set of C-language user defined functions for Postgres that allow client connections to a Gearman server.
You can try out the pgGearman 0.1 release on Launchpad!
postgres
CSS @font-face in Firefox 3.5
By Jon Jensen
July 15, 2009
This has been frequently mentioned around the web already, but it’s important enough that I’ll bring it up again anyway. Firefox 3.5 adds the CSS @font-face rule, which makes it possible to reference fonts not installed in the operating system of the browser, just as is done with images or other embedded content.
Technically this is not a complicated matter, but font foundries (almost all of whom have a proprietary software business model) have tried to hold it back hoping for magical DRM to keep people from using fonts without paying for them, which of course isn’t possible. As one of the original Netscape developers mentioned, if they had waited for such a thing for images, the web would still be plain-text only.
The quickest way to get a feel for the impact this change can have is to look at Ian Lynam & Craig Mod’s article demonstrating @font-face in Firefox 3.5 side-by-side with any of the other current browsers. It is exciting to finally see this ability in a mainstream browser after all these years.
browsers css