Aligning monospace font text columns with an old Unix tool

2022-05-30T00:00:00+00:00

Photo by Garrett Skinner

A blast from 1990: column

A while back I learned of a nice old Unix command-line tool called column. It first appeared in 4.3BSD-Reno, released in July 1990. (This is not to be confused with the different, even older Unix tool col.)

column formats plain text into nice columns based on the width of the input separated by tabs or groups of spaces.

For example, take this mess found in a server’s /etc/fstab file defining filesystem mount points. It is a real example lightly redacted to remove business details. You may need to scroll right to see the end of the fairly long lines:

/data3/customer_uploads   /home/interch/htdocs/shared/customer_uploads none    rw,bind 0   0
/data3/customer_images    /home/interch/htdocs/shared/customer_images  none    rw,bind 0   0
/data3/images/items       home/interch/htdocs/images/items     none    rw,bind 0       0
/data3/images/thumb       home/interch/htdocs/images/thumb     none    rw,bind 0       0
/data3/upload_images       /home/interch/upload_images     none    rw,bind 0       0
/data3/design    /home/interch/htdocs/shared/design  none    rw,bind 0   0
/data3/design_temp   /home/interch/htdocs/shared/design_temp none    rw,bind 0   0
/data3/reports/cat1       /home/interch/htdocs/cat-numero-uno/images/reports     none    rw,bind 0       0
/data3/reports/cat2       /home/interch/htdocs/cat-zahl-zwei/images/reports     none    rw,bind 0       0
/data3/reports/cat3	/home/interch/htdocs/cat-number-three/images/reports	none	rw,bind	0	0
/data3/reports/cat4    /home/interch/htdocs/cat-quatre/images/reports     none    rw,bind 0       0
/data3/reports/cat5    /home/interch/htdocs/cat-pět/images/reports     none    rw,bind 0       0
/data3/shared_var /home/interch/catalogs/shared/var none    rw,bind 0 0

That is really unsightly and includes a mix of spaces and tabs.

If we feed it to column -t we get the same data aligned much nicer:

% column -t /etc/fstab
/data3/customer_uploads  /home/interch/htdocs/shared/customer_uploads          none  rw,bind  0  0
/data3/customer_images   /home/interch/htdocs/shared/customer_images           none  rw,bind  0  0
/data3/images/items      home/interch/htdocs/images/items                      none  rw,bind  0  0
/data3/images/thumb      home/interch/htdocs/images/thumb                      none  rw,bind  0  0
/data3/upload_images     /home/interch/upload_images                           none  rw,bind  0  0
/data3/design            /home/interch/htdocs/shared/design                    none  rw,bind  0  0
/data3/design_temp       /home/interch/htdocs/shared/design_temp               none  rw,bind  0  0
/data3/reports/cat1      /home/interch/htdocs/cat-numero-uno/images/reports    none  rw,bind  0  0
/data3/reports/cat2      /home/interch/htdocs/cat-zahl-zwei/images/reports     none  rw,bind  0  0
/data3/reports/cat3      /home/interch/htdocs/cat-number-three/images/reports  none  rw,bind  0  0
/data3/reports/cat4      /home/interch/htdocs/cat-quatre/images/reports        none  rw,bind  0  0
/data3/reports/cat5      /home/interch/htdocs/cat-pět/images/reports           none  rw,bind  0  0
/data3/shared_var        /home/interch/catalogs/shared/var                     none  rw,bind  0  0

That isn’t just prettier! It also makes some things stand out prominently at a glance:

In the second column there are two mount points not starting with /.
It’s easy to see that most of the paths in the second column start with /home/interch/htdocs/ and the few that don’t, stand out.
The final 4 columns are all identical, which was unclear in the unaligned original.

Text tables to JSON

The util-linux version of column includes extra options beyond the original. One very useful one is -J or --json which produces a JSON object for each row based on column names you define in the argument -N or --table-columns.

Reviewing man fstab for details on what each column is used for in the sample above, we can instruct column to produce JSON output like this:

% column -J -n fstab -N spec,file,vfstype,mntops,freq,passno /etc/fstab

Which gives us this result:

{
   "fstab": [
      {
         "spec": "/data3/customer_uploads",
         "file": "/home/interch/htdocs/shared/customer_uploads",
         "vfstype": "none",
         "mntops": "rw,bind",
         "freq": "0",
         "passno": "0"
      },{
         "spec": "/data3/customer_images",
         "file": "/home/interch/htdocs/shared/customer_images",
         "vfstype": "none",
         "mntops": "rw,bind",
         "freq": "0",
         "passno": "0"
      },{
         "spec": "/data3/images/items",
         "file": "home/interch/htdocs/images/items",
         "vfstype": "none",
         "mntops": "rw,bind",
         "freq": "0",
         "passno": "0"
      },{
         "spec": "/data3/images/thumb",
         "file": "home/interch/htdocs/images/thumb",
         "vfstype": "none",
         "mntops": "rw,bind",
         "freq": "0",
         "passno": "0"
      },{
         "spec": "/data3/upload_images",
         "file": "/home/interch/upload_images",
         "vfstype": "none",
         "mntops": "rw,bind",
         "freq": "0",
         "passno": "0"
      },{
         "spec": "/data3/design",
         "file": "/home/interch/htdocs/shared/design",
         "vfstype": "none",
         "mntops": "rw,bind",
         "freq": "0",
         "passno": "0"
      },{
         "spec": "/data3/design_temp",
         "file": "/home/interch/htdocs/shared/design_temp",
         "vfstype": "none",
         "mntops": "rw,bind",
         "freq": "0",
         "passno": "0"
      },{
         "spec": "/data3/reports/cat1",
         "file": "/home/interch/htdocs/cat-numero-uno/images/reports",
         "vfstype": "none",
         "mntops": "rw,bind",
         "freq": "0",
         "passno": "0"
      },{
         "spec": "/data3/reports/cat2",
         "file": "/home/interch/htdocs/cat-zahl-zwei/images/reports",
         "vfstype": "none",
         "mntops": "rw,bind",
         "freq": "0",
         "passno": "0"
      },{
         "spec": "/data3/reports/cat3",
         "file": "/home/interch/htdocs/cat-number-three/images/reports",
         "vfstype": "none",
         "mntops": "rw,bind",
         "freq": "0",
         "passno": "0"
      },{
         "spec": "/data3/reports/cat4",
         "file": "/home/interch/htdocs/cat-quatre/images/reports",
         "vfstype": "none",
         "mntops": "rw,bind",
         "freq": "0",
         "passno": "0"
      },{
         "spec": "/data3/reports/cat5",
         "file": "/home/interch/htdocs/cat-pět/images/reports",
         "vfstype": "none",
         "mntops": "rw,bind",
         "freq": "0",
         "passno": "0"
      },{
         "spec": "/data3/shared_var",
         "file": "/home/interch/catalogs/shared/var",
         "vfstype": "none",
         "mntops": "rw,bind",
         "freq": "0",
         "passno": "0"
      }
   ]
}

JSON takes a lot more room, and of course is not the format Linux expects for this particular file, but transforming tabular data to JSON format in other situations can be more readable for exchange across different systems since each field is labeled, and unused fields can be omitted. Plus JSON syntax is rigorously defined, nested data structures are possible, etc.

Columnizing lists

All versions of column can also columnize lists, either horizontally (across) or vertically (down). Take this list of people’s names:

Amy
Anna
Bob
Brenda
Cameron
Doug
Emily
Frank
Jane
Jill
Jim
Joe
John
Karen
Kate
Liz
Mary
Mike
Sarah
Steve
Victoria

Put that in file /tmp/names and column will format it in columns fitting the width of your terminal:

% column /tmp/names
Amy		Cameron		Jane		John		Mary		Victoria
Anna		Doug		Jill		Karen		Mike
Bob		Emily		Jim		Kate		Sarah
Brenda		Frank		Joe		Liz		Steve

It uses 2 or more tab characters to separate the columns, based on standard terminal 8-space tab stops, so the above doesn’t look right here on the web.

What appears in my terminal looks like:

% column /tmp/names
Amy             Cameron         Jane            John            Mary            Victoria
Anna            Doug            Jill            Karen           Mike
Bob             Emily           Jim             Kate            Sarah
Brenda          Frank           Joe             Liz             Steve

Or you can use column -t that we discussed earlier to format the columns more compactly with spaces:

% column /tmp/names | column -t
Amy     Cameron  Jane  John   Mary   Victoria
Anna    Doug     Jill  Karen  Mike   
Bob     Emily    Jim   Kate   Sarah  
Brenda  Frank    Joe   Liz    Steve

You can also ask for the list to be delivered horizontally, rather than vertically, with column -x:

% column -x /tmp/names | column -t
Amy    Anna   Bob       Brenda  Cameron  Doug
Emily  Frank  Jane      Jill    Jim      Joe
John   Karen  Kate      Liz     Mary     Mike
Sarah  Steve  Victoria

Note that it isn’t doing anything to affect your ordering. You can order the lines in your original file however you want and it will preserve them. But other tools can help you here: Use sort -u to sort alphabetically and remove duplicates.

For more options and details see the column man page for the util-linux column version or FreeBSD column version (same as macOS).

A blast from 1974: pr

There is an even older Unix tool for columnizing lists in the same way. It is called pr and dates to First Edition Unix in 1971, but did not gain the options we are using here until Fifth Edition Unix in 1974 as seen in the V6 pr man page.

We need to tell it how many columns to produce, so we will ask for 6 columns as column was doing above. Note that pr emits a curious mix of tabs and spaces, which cat -t reveals here as ^I (since a tab is the same thing as Control+I):

% pr -t -6 /tmp/names | cat -t
Amy^I    Cameron^IJane^I    John^ILiz^I    Sarah
Anna^I    Doug^IJill^I    Karen^IMary^I    Steve
Bob^I    Emily^IJim^I    Kate^IMike^I    Victoria
Brenda^I    Frank^IJoe

But in a terminal it looks fine:

% pr -t -6 /tmp/names
Amy         Cameron     Jane        John        Liz         Sarah
Anna        Doug        Jill        Karen       Mary        Steve
Bob         Emily       Jim         Kate        Mike        Victoria
Brenda      Frank       Joe

See the many more options of pr in the GNU coreutils pr man page and FreeBSD pr man page (same as macOS).

A blast from 1979: expand

A useful tool for dealing with tabs is expand, which first appeared in 3BSD in 1979. (Despite the FreeBSD and macOS man pages saying it appeared in 1BSD, I don’t see it there or in 2BSD.)

We can use it to convert tabs to spaces just like a terminal would:

% pr -t -6 /tmp/names | expand | cat -t    
Amy         Cameron     Jane        John        Liz         Sarah
Anna        Doug        Jill        Karen       Mary        Steve
Bob         Emily       Jim         Kate        Mike        Victoria
Brenda      Frank       Joe

expand accepts the optional -t argument with a number to use as tabstop width instead of the default 8.

And unexpand

I have used expand for years, but only recently learned of unexpand which goes the other way, converting runs of spaces into tabs:

% pr -t -6 /tmp/names | expand | unexpand -a         
Amy         Cameron     Jane        John        Liz         Sarah
Anna        Doug        Jill        Karen       Mary        Steve
Bob         Emily       Jim         Kate        Mike        Victoria
Brenda      Frank       Joe

It looks fine in the terminal, but let’s see if it actually used tabs:

% pr -t -6 /tmp/names | expand | unexpand -a | cat -t
Amy^I    Cameron^IJane^I    John^ILiz^I    Sarah
Anna^I    Doug^IJill^I    Karen^IMary^I    Steve
Bob^I    Emily^IJim^I    Kate^IMike^I    Victoria
Brenda^I    Frank^IJoe

Interesting… For some reason unexpand doesn’t actually convert all the spaces to tabs, but just one initial tab per gutter between columns. Running the output through unexpand -a again has no further effect. Strange.

The GNU coreutils version of unexpand that lives in most Linux systems is what I would call bug-compatible with the BSD version in this regard.

Oh, well. There’s probably a reason.

Since it has handled turning the initial tabstop’s varying number of spaces into a tab, we can easily remove the remaining fixed multiples of spaces on our own for a more compact list:

% pr -t -6 /tmp/names | expand | unexpand -a | sed 's/    //g'         
Amy     Cameron Jane    John    Liz     Sarah
Anna    Doug    Jill    Karen   Mary    Steve
Bob     Emily   Jim     Kate    Mike    Victoria
Brenda  Frank   Joe

Ready at hand

column, pr, expand, and unexpand all come with most Linux and BSD systems, including macOS. It is amazing what great old tools are on many of our computers all the time, waiting to be used!

You can use these programs as filters inside your favorite text editor or IDE. For example, to achieve the same list columnization as above you can select a block of text and send it to external commands:

In Vim, visually select text with v or V and then type !column | expand and press Enter.
In VS Code you can install an extension called Filter Text (by yhirose). Once installed, type Control+K Control+F (⌘K ⌘F on macOS) and then column | expand and Enter.
In IntelliJ IDEA you can install a plugin called Shell Filter (by Dennis Plöger). Once installed, select your text, then choose menu item Edit > Custom Shell Filter and then type column -c 80 | expand and Enter.
Most other editors have a way to do this too. Search for “filter”, “pipe”, and/or “command”.

Your selection will be replaced by the output.

Unicode

These tools come from an era when one byte of input resulted in one visual character, so for any UTF-8 characters outside the limited classic ASCII character set, we could expect old tools may miscalculate the needed space between columns.

Originally in my testing they appeared not to be aware of some Unicode characters’ width, perhaps due to combining characters or alternate forms. But surprisingly they have been modernized and seem to work fine with most Unicode characters such as Latin letters with diacritics, letters in other alphabets, Chinese characters, and even emoji! 😊

EditorConfig: Ending the Spaces vs. Tabs Confusion

2022-04-30T00:00:00+00:00

Photo by Garrett Skinner

Varieties of text formatting

Most everyone who has worked on a software development project with a group of other people has encountered the problem of source code being formatted in different ways by different text editors, IDEs, and operating systems.

The main variations go back to the 1970s or earlier, and include the questions:

Will indentation be done by tabs (an ASCII control character) or spaces?
- If indentation is done by spaces, how many spaces are used for each indentation level?
What will indicate the end of each line (EOL)? The choices are:
- a line feed (LF), used by the Unix family including Linux and modern macOS
- a carriage return (CR), used by old pre-Unix Macintosh and some now-obscure operating systems
- both together (CRLF) used by Windows and most Internet protocols
Which character set encoding will be used? Common choices are:
- Unicode UTF-8 encoding, used by Linux, macOS, and most other Unixes, and standard on the Internet
- Unicode UTF-16 encoding (with either little-endian or big-endian encoding), used by modern Windows
- legacy ISO-8859 and Windows “code page” encodings in older documents and codebases

Editor configurations in conflict

Causing widespread frustration, by default, text editors and IDEs generally are each configured differently, and once set, the choices apply broadly from then on. But each developer can simply configure their editor to follow their team’s standards, right? Well, maybe.

First, getting that to happen for every developer and every different editor being used isn’t straightforward. It typically requires a document showing instructions and/or screenshots of how to configure each editor. It may have to be redone after a major upgrade or move to a new computer.

Second, and often a more persistent problem, standards may vary across different projects and even for different types of files within a given project. Ruby code is typically indented with 2 spaces, while perhaps in your project JavaScript uses 4 spaces and HTML uses tabs.

If you start a new project from scratch you can probably settle on a single standard, but in existing large codebases, it can make a lot of version control change “noise” to mess with that.

Computers are good at keeping track of lots of little details, so isn’t there some way to have the computer deal with this?

Storing configuration in the project

What if we store the text editor’s or IDE’s configuration in the project instead of per user, so it can go with the project to each new developer and tell their editor how to behave?

For many years that has been possible with some editors, but the configuration had to be set up separately for each editor, and often the feature is disabled by default.

Let’s consider the two most popular terminal-based editors on Unix, partisans in a long-running editor war:

Vim

Vim has a feature called a “modeline” that allows for configuration settings to appear within the top or bottom 5 lines of the file.

For example, to instruct Vim to use spaces instead of tabs and 4-space tab stops, we can add to the top or bottom of our C source code file:

/* vim: tabstop=4 shiftwidth=4 expandtab
 */

Since it gets tedious putting those special configuration comments in each file, Vim has an option to read a .vimrc file from the current directory, which applies to all files there and can be committed to version control.

This feature is disabled by default because Vim has in the past been vulnerable to files with malicious settings running arbitrary code.

You can :set exrc secure to enable the modeline feature in a code base you trust, and also to restrict what it can do.

Emacs

In Emacs the same thing can be done on the first or second line of the file. (Of course its setting names differ from Vim’s.) For example consider this configuration in C source code:

/* -*- mode: c; indent-tabs-mode: nil; c-basic-offset: 4; tab-width: 4 -*- */

Alternately you can use “Local Variables” set at the end of the file in as many lines as needed:

/* Local Variables:      */
/* mode: c               */
/* indent-tabs-mode: nil */
/* c-basic-offset: 4     */
/* tab-width: 4          */
/* End:                  */

Emacs also has “Directory Variables” that can be set in the file .dir-locals.el for a directory and its subdirectories.

Others

Even if someone has gone to the trouble to set up such editor configuration files and add them to the project code repository, how often has that been done for your editor or IDE?

And how often is one out of sync with the others?

This is not the way to success.

EditorConfig to the rescue

About 10 years ago Trey Hunner and Hong Xu shared with the world EditorConfig, their creation to solve this problem across ideally all editors.

They intentionally kept EditorConfig fairly limited in scope. It covers a limited number of the most important editor options so that the standard would be simple enough to be implemented for every editor either internally or as a plugin, and there would be no arbitrary code execution possible to cause security problems.

In EditorConfig the configuration for our examples and hypotheticals above lives in a .editorconfig file in the root of the project that looks like this:

# top-most EditorConfig file
root = true

# basics for all files in our project
[*]
charset = utf-8
end_of_line = lf

# C and JavaScript source get 4-space indents
[*.{c,js}]
indent_style = space
indent_size = 4

# Ruby gets 2-space indents
[*.rb]
indent_style = space
indent_size = 2

# HTML gets tab indents
[*.html]
indent_style = tab

In a big project you may want to have separate, smaller .editorconfig files in different directories. You can omit the root = true setting in subdirectories to inherit settings from the top-level .editorconfig file.

There are a couple of other options that are nice to specify.

This one removes any tabs or spaces from the end of lines:

trim_trailing_whitespace = true

Those are rarely needed or semantically meaningful, so it’s nice to remove them. But there are a few cases where they can matter such as in Markdown.

This one determines whether the last line in the file will end with a newline:

insert_final_newline = true

By default some editors add to the last line a newline (such as Vim) and some don’t (such as Emacs), leading to needless changes as various developers change files.

Typically every line should end with a newline, so that’s a good editor feature to enable. But you could have some text template that should not end with a newline, so might need to specify false for that type of file.

And those are most of the features of EditorConfig! The file format details are easy to digest.

Editor & IDE support

EditorConfig is now widely supported. These popular editors & IDEs recognize .editorconfig files with no extra work:

IntelliJ IDEA and most of its language-specific variants
GitHub
GitLab
Visual Studio
BBEdit
and others

And these support it with a plugin:

VS Code
Vim
Emacs
Sublime Text
TextMate
Eclipse
Atom
Notepad++
Geany
and others

The plugins are typically easy to install system-wide from your operating system’s package manager, or else locally for your user only.

Do you need it?

Yes, I think you do.

I know of no reason for any developer not to use EditorConfig, in every editor, for every project. It’s simple and at long last solves this small set of problems well.

One possible counterargument: If, before every version control commit, you run an automatic code formatter such as Prettier (in Node.js, for many languages) or a language-specific one such as gofmt, rustfmt, etc., you could perhaps live without your editor knowing how your files should be saved.

But isn’t it better if your editor knows what kind of line endings and indents to use, rather than waiting for a code formatter to correct such fundamental things after you save? It is easy to start with a single .editorconfig file long before you have a continuous integration set up for the project.

And many projects don’t format code automatically, and instead just “lint” it to report on deviations from the project standards. But that requires work to correct, and can be ignored if not enforced.

Many open source projects large and small use EditorConfig, including this blog itself. But in recent months I have found several developers who had not yet heard of EditorConfig, so I want to spread awareness of it. I hope you’ll use EditorConfig too!

Formatting SQL code with pgFormatter within Vim

2022-04-26T00:00:00+00:00

Photo by Garrett Skinner

Sometimes a little, seemingly simple tip can make a world of difference. I’ve got enough gray hair these days that it would be pretty easy for me to start thinking I’d seen an awful lot, yet quite frequently when I watch a colleague working in a meeting or a tmux session or somewhere, I learn some new and simple thing that makes my life demonstrably easier.

Luca Ferrari recently authored a post about using pgFormatter in Emacs; essentially the same thing works in Vim, my editor of choice, and it’s one of my favorite quick tips when working with complicated queries. I don’t especially want to get involved an editor war, and offer the following only in the spirit of friendly cooperation for the Vim users out there.

As Luca mentioned, pgFormatter is a convenient way to make SQL queries readable, automatically. It’s easy enough to feed it some SQL, and get a nice-looking result as output:

$ pg_format < create_outbreaks.sql
INSERT INTO outbreak                      
SELECT                              
    nextval('outbreak_id'::regclass),
    extract('year' FROM now())::text || '-' || nextval('outbreak_number_seq')::text, --number
    (                        
        SELECT  
            first_name
        FROM person TABLESAMPLE BERNOULLI (10)
LIMIT 1), -- name
    NOW() - interval '1 day' * random() * 100, (
        SELECT
            id
        FROM "user" TABLESAMPLE BERNOULLI (10)
...

In my perfect world I might quibble with some of its formatting decisions, such as the lack of indent on the LIMIT 1 line above. But in practice the results are good enough for my tastes that I haven’t bothered to investigate whether I can improve them. I just use it, and it’s good enough for me.

And because Vim lets me highlight a region, pipe it through an external program, and replace the region with that program’s output, it’s easy to use it simply by selecting a section of code and typing :!pg_format like this:

Linting Ruby In Your Editor

2019-06-27T00:00:00+00:00

Photo by Kimberly Vardeman, used under CC BY 2.0

Ruby developers have access to a variety of linters and static program analysis tools that can greatly improve developer efficiency and code quality by catching syntax errors, detecting code smells, and making coding style suggestions based on popular style guides.

I have been a long-time advocate of configuring development environments with automatic code linting and will use this post to highlight some of the available tools for Ruby and methods for integrating them with popular code editors (Vim, Emacs, and Visual Studio Code).

Configuring your editor for automatic linting makes it much easier to identify and fix issues with your code at development time, and in-editor integration is very convenient for highlighting problems as you type (or save), making it easy to evaluate and improve the quality of your code as it is written.

Three popular linting plugins/extensions for Vim, Visual Studio Code, and Emacs are:

Asynchronous Lint Engine (ALE) Plugin for Vim

Provides asynchronous linting in Vim while you edit, and displays warnings/error messages in the editor. Supports the following tools for Ruby development and runs them automatically if they are found in your PATH:

Ruby Extension for Visual Studio Code

Requires configuring the settings JSON file to enable each tool on an individual basis:

// Basic settings: turn linter(s) on
"ruby.lint": {
	"reek": true,
	"rubocop": true,
	"ruby": true, //Runs ruby -wc
	"fasterer": true,
	"debride": true,
	"ruby-lint": true
},

debride
fasterer
reek
rubocop
ruby -wc (verbose syntax check)
ruby-lint (unmaintained)

Flycheck Extension for Emacs

Detects and uses the following tools when editing Ruby code:

reek
rubocop
ruby -wc (verbose syntax check)
ruby-lint (unmaintained)

Linter Selection

I suggest starting small and only installing one or two linters to begin with; some choices provide similar features and will display conflicting or redundant warnings/errors if used at the same time. I think that RuboCop is a great first choice, and I have spent years using it as my primary linter, though recently I have started supplementing it with Reek and Fasterer.

I highly recommend the use of these tools and consider them essential for any Ruby developer that is interested in improving the quality, reliability, and maintainability of their code.

Vim Golf: Learning New Skills for Code Editors

2016-11-28T00:00:00+00:00

Vim logo freely licensed from Wikimedia Commons

Vim is a text-based editor that has been around for 25 years. It comes pre-installed on Linux distributions, so it is a great tool for developing on servers. One of the advantages of Vim is that oft-used keystrokes can be performed without moving your hands from the keyboard (there is no mouse in Vim).

Many of the engineers here at End Point use Vim for our daily development work, and recently, a few of us got together online to try to learn some new tricks and tips from each other. Being efficient with Vim not only improves productivity, it’s a lot of fun.

Similar to playing a round of golf, we tested each other with various editing tasks, to see who could perform the task in the fewest number of keystrokes. This is known as “Vim Golf.” There is even an entire website devoted to this.

In this post, we share some of the interesting tricks that were shown, and also some links to further learning about Vim.

Tips & Tricks

Indenting text: there are multiple ways to do this, but a few are:
- Visually-select the lines of text to indent (Ctrl v or Shift v), then > to indent, or < to outdent. Press . to perform this action again and again.
- Locate the line numbers for the lines you wish to change (:set number to turn on line numbering), then :17,36» to indent lines 17-36 two indentation levels.
- Define width of a tab :set tabstop=4 would for example set a tab to 4 spaces.
- Use spaces defined in tabstop instead of an actual tab character (^I) when the Tab key is pressed :set expandtab or :set et
- Replace tab settings for current line :retab
- Replace tab settings for current document :retab!
Visually-selecting text: Ctrl v will perform a visual column selection, while Shift v will do a row selection.
:set will show all the currently-set options.
For paging up or down, use Ctrl b and Ctrl f. You can also use PgUp and PgDn keys if you want to move your hands a bit :-)
Moving the cursor around the page:
- Type M to move the cursor to the middle of the screen
- Type H to move the cursor to the top of the screen
- Type L to move the cursor to the bottom of the screen
- Type gg to move the cursor to the top of the document
- Type G to move the cursor to the bottom of the document
Moving the page around the cursor:
- Type zz to make the current position float to the middle of the screen
- Type zt to make the current position float to the top of the screen
- Type zb to make the current position float to the bottom of the screen
Search and replace:
- Find and replace all instances of a string: %s/find_this/replace_with_this/g
- Case-insensitive find and replace all instances of a string: %s/find_this/replace_with_this/gi
- Find then ask confirmation before replacing: %s/find_this/replace_with_this/c
- Search history: Vim maintains search history which is easy to access using / or ? then navigation through the list using the up and down arrows.
Deleting from the current position to the bottom of the file: dG
Jumping to the first position in the current line: 0
Find the next occurrence of a character in the current line: f then the character. To search backwards, use F
Undo a command: u (run this multiple times for additional undo steps)
Redo your undo: Ctrl r
Travel back in time to see the document as it was 30 mins ago :earlier 30m then revert with :later 30m
Reselect the last visual selection gv
Run a system command from within Vim :! [command]
Review your previous vim command history q:

For Further Learning

Built-in Vim help: use :h name_of_command_or_setting (e.g. :h f)
View manpage for the word under the cursor: K
Cheat sheets
Vim Tips with Ben podcast
Vimcasts — free screencasts
Tweets from Mastering Vim

With contributions from Sam Batschelet

Report on The Perl Conference 2016

2016-07-27T00:00:00+00:00

In June, I traveled to Orlando, Florida to attend the event formerly known as Yet Another Perl Conference (or YAPC::NA), now known as The Perl Conference. This was my second time in a row to attend this conference (after my first attendance back in 2007).

Conferences are a great place to learn how others are using various tools, hear about new features, and interact with the community. If you are speaking, it’s a great opportunity to brush up on your subject, which was true for me in the extreme, as I was able to give a talk on the PostgreSQL database, which I hadn’t used in a long time (more on that later).

The conference name

The event organizers were able to license the name The Perl Conference from O’Reilly Media, as O’Reilly doesn’t hold conferences by this name anymore. This name is now preferred over “YAPC” as it is more friendly to newcomers and more accurately describes the conference. More on the name change.

Notes from the conference

Over the three days of the conference, I was able to take in many talks. Here are some of my more interesting notes from various sessions:

Yes, this is happening @YAPCNA

— Jason Hall (@JayceHall) June 22, 2016

MetaCPAN is the best way to browse and search for Perl modules. Anyone can help with development of this fine project, via their GitHub.
Ricardo Signes says “use subroutine signatures!” They are “experimental”, but are around to stay.
Perl6 is written in Perl6 (and something called “Not Quite Perl”). This allows one to read the source to figure out how something is done. (There were many talks on Perl6, which is viewed as a different programming language, not a replacement for Perl5.)
jq is a command-line utility that can pretty-print JSON (non Perl, but nice!)
Ricardo Signes gave a talk on encoding that was over my head, but very interesting.
The presenter of Emacs as Perl IDE couldn’t attend, so Damian Conway spoke on VIM as Perl IDE (photo above)

From John Anderson’s talk:

Just say “no” to system Perl. Use plenv or the like.
There’s a DuckDuckGo bang command for searching MetaCPAN: !cpan [module]
Use JSON::MaybeXS over the plain JSON module.
Use Moo for object-oriented programming in Perl, or Moose if you must.
Subscribe to Perl Weekly
Submit your module on PrePAN first, to receive feedback before posting to CPAN.

Lee Johnson gave a talk called Battling a legacy schema with DBIx::Class (video). Key takeaways:

When should I use DBIC?
Something that has grown organically could be considered “legacy,” as it accumulates technical debt
With DBIC, you can start to manage that debt by adding relationships to your model, even if they aren’t in your database
RapidApp’s rdbic can help you visualize an existing database

D. Ruth Bavousett spoke on Perl::Critic, which is a tool for encouraging consistency. Basically, Perl::Critic looks at your source code, and makes suggestions for improvement, etc. These suggestions are known as “policies” and can be configured to enable or disable any of them, or to even write new policies. One suggestion was to create a Git hook to run the perlcritic command at the time code is committed to the source code repository (possibly using App::GitHooks). End Point has its own perlcritic configuration, which I have started trying to use more.

Logan Bell shared Strategies for leading a remote team. Some of the tools and techniques he uses include:

tmate for terminal sharing
HipChat, with a chat room just for complaining called “head to desk”
Holds one-on-one meetings every Monday for those he works with and directs
Has new team members work on-site with another team member their first week or so, to help understand personalities that don’t often come across well over chat
Tries to have in-person meetings every quarter or at least twice a year, to bring the team together

My talk

Finally, my own talk was titled Stranger in a Strange Land: PostgreSQL for MySQL users (video). I hadn’t used Postgres in about seven years, and I wanted to get re-acquainted with it, so naturally, I submitted a talk on it to spur myself into action!

In my talk, I covered:

the history of MySQL and Postgres
how to pronounce “PostgreSQL”
why one might be preferred over the other
how to convert an existing database to Postgres
and some tools and tips.

I enjoyed giving this talk, and hope others found it helpful. All in all, The Perl Conference was a great experience, and I hope to continue attending in the future!

All videos from this year’s conference

Vim Plugin Spotlight: CtrlP

2015-02-06T00:00:00+00:00

When I started using Vim I relied on tree-based file browsers like netrw and NerdTree for navigating a project’s files within the editor. After discovering and trying the CtrlP plugin for Vim I found that jumping directly to a file based on its path and/or filename could be faster than drilling down through a project’s directories one at a time before locating the one containing the file I was looking for.

After it’s invoked (usually by a keyboard shortcut) CtrlP will display a list of files in your current project and will filter that list on the fly based on your text input, matching it against both directory names and file names. Pressing with CtrlP open toggles through two other modes: most recently used files, and current buffers. This is useful when you want to narrow down the list of potential matches to only files you have worked with recently or currently have open in other buffers. I use CtrlP’s buffer mode to jump between open files so often that I added a custom mapping to invoke it in my .vimrc file:

map <leader>b :CtrlPBuffer<cr></cr>

CtrlP has many configuration options that can affect its performance and behavior, and installing additional plugins can provide different matcher engines that search through a directory more quickly and return more relevant results than the default matcher. Alternate matchers I’ve found include:

Of these, I’ve had the best luck with FelixZ’s ctrlp-py-matcher. It’s easy to install, works on most systems without requiring additional dependencies, and manages to be both faster and return more relevant results than the built-in CtrlP matcher.

CtrlP is well documented in both its README (available on its GitHub project page) and its Vim documentation (available with :help ctrlp within Vim). The documentation covers the different commands and configuration options provided by CtrlP but simply installing the plugin and hitting on your keyboard is enough to get you started with a faster way to navigate between files in any codebase.

vim-airline: A lightweight status/tabline for Vim

2014-06-02T00:00:00+00:00

My standard Vim configuration makes use of around 30 different plugins and I consider vim-airline to be one of the most indispensable because of its built-in functionality and superb integration with a variety of other Vim plugins. It’s a great starting point for anyone looking to extend their Vim setup with additional plugins.

I became interested in vim-airline the first time I saw screenshots of it; the color schemes, custom glyphs[1] and indicators immediately revealed value beyond the basic status bar that a stock Vim installation provides. After installing and spending some time using vim-airline I discovered additional benefits due to its integration with other plugins such as Fugitive, Syntastic and CtrlP. vim-airline provides a common platform for integrating the display indicators of plugins from various authors into one view and presents them all with consistent a consistent style.

A quick comparison of Vim with vim-airline installed:

vs. a standard Vim installation:

reveals new indicators for the current Vim mode, git branch, open buffers, and line endings. Integrating other plugins can add additional indicators for syntax errors, trailing whitespace, and more.

Using vim-airline also helped me adopt the usage of built-in Vim functionality that I had previously overlooked. I had not used Vim’s tab features, favoring a single split window, but after enabling vim-airline’s tabline feature I am now in the habit of using a combination of tabs and split windows to organize my workspace and have found that I am more easily able to keep track of multiple files at a time.

I recommend reading through the documentation for vim-airline, trying it out and then installing some of the other plugins that it integrates with in order to develop your own preferred set of Vim plugins. I spend most of my workday in a Vim session and consider it a good investment to research different plugins and features that can increase productivity and developer happiness.

[1] Note that some of the nice-looking but non-standard characters available in vim-airline require the use of a patched font; pre-patched versions of many popular monospaced fonts are available in the powerline-fonts repository on GitHub and are easy to install (I’m a fan of Adobe’s Source Code Pro font).

Highlighting Search Pattern Matches in Vim

2013-12-30T00:00:00+00:00

Vim’s hlsearch option is a commonly-used way to enable visual feedback when searching for patterns in a Vim buffer. When highlighting of search matches is enabled (via :set hlsearch), Vim will add a colored background to all text matching the current search.

Search highlighting is useful but it’s not obvious how to turn off the highlighting when you’re no longer concerned with the results of your last search. This is particularly true if you didn’t enable the hlsearch setting yourself but inherited it from a prebuilt package like Janus or copied someone else’s .vimrc file.

One commonly-used way to clear the highlighting is to search for a garbage string by doing something like /asdfkj in normal mode. This method will clear the search highlights but has the undesired side effect of altering your search history.

A better way to disable search highlighting temporarily is with the :nohlsearch command (which can be abbreviated to :noh). This will clear the highlights temporarily, but they’ll be turned back on the next time you perform a search. Also, you can use the n/N keys to resume your previous search, which isn’t possible if you use the above method of searching for a garbage string.

For more information on highlighting search matches in Vim, check out the Highlight all search pattern matches entry on the Vim Tips Wiki.

Vim tabs and splits

2013-07-15T00:00:00+00:00

Vim is my go-to code editor these days. After many years of using different editors, I’ve settled on Vim as my editor of choice. There are some things I’ve done to make using Vim more enjoyable and usable and this blog post is dedicated to some of those things that I use everyday.

Tabs

I love using tabs in Vim. I know there are some who are opposed to tabs, but I find them invaluable. The default shortcuts for manipulating tabs are a little cumbersome, which I believe deters many users. Here are some modifications that I added to my vimrc.

nmap <silent> tt :tabnew<CR>
nmap <silent> [g :tabprevious<CR>
nmap <silent> ]g :tabnext<CR>
nmap <silent> [G :tabrewind<CR>
nmap <silent> ]G :tablast<CR>

First, I’m using nmap here which says to only map these keys in normal mode. Next, I use which keeps my editor clean of any distractions while performing the task. I find that the double tap short-cuts (see more below) tt work really well for normal mode and I love their simplicity. Double-tap t and you have a new tab. Using the bracket navigation is something that I’ve stolen from Tim Pope’s vim-unimpaired plugin. Using g and G work for me, but you can use whatever you like.

Splits

I believe most Vim users use splits often. At first, I found that splitting my current buffer was also cumbersome using the default method in Vim (:split and :vsplit). So utilizing the double-tap method I used for tabs, I created my own shortcuts:

nmap <silent> vv :vsp<CR>
nmap <silent> ss :sp<CR>

Once you have a split, navigating between them can be a little bit of a pain as well. Here’s my optimization:

map <C-h> <C-w>h
map <C-j> <C-w>j
map <C-k> <C-w>k
map <C-l> <C-w>l

After using Vim’s splitting capability for a while, I noticed I didn’t always like where the split occurred. Sometimes above, sometimes to the left. To ensure that you have consistency, try these settings:

set splitright " When splitting vertically, split to the right
set splitbelow " When splitting horizontally, split below

Using splits and tabs give you a lot of flexibility in managing the code you are currently working. Adding just a few shortcuts and optimizations to your vimrc can really speed up your workflow and help you navigate your code more quickly.

One more thing

Want a quick way to get to your vimrc?

:e $MYVIMRC

Vim — working with encryption

2012-05-16T00:00:00+00:00

On occasion I have to work with encrypted files for work or personal use. I am partial to a Linux environment and I prefer Vim as my text editor, even when I am only reading a file. Vim supports quite a few different ways of interfacing with external encryption packages. I only use two of those variations as described below.

Vim comes packaged with a default encryption mechanism referred to as VimCrypt in the documentation. I typically use this functionality as a temporary solution in a situation where my GPG is not immediately available, like a remote system that is not mine.

Using Vim’s default VimCrypt feature

Creating a new encrypted file or open a plain text file you wish to encrypt:

vim -x <filename></filename>

This will create a new file if it does not exist or open an existing file and then prompt you for a password. This password is then used as the key to encrypt and decrypt the specified file. Upon saving and exiting this file, it will be saved in this encrypted format using your crypt key.

You can also save and encrypt an open file you are currently working on like so. Please note this is a capital X:

:X

This will also ask you for a password to encrypt the file.

Reasons I usually don’t use this option:

Vim uses a weak encryption method by default. Vim encrypts the file using an encryption method ‘zip’, the same encryption algorithm that is used by Pkzip (known to be flawed). You can set the default encryption to use the more secure ‘blowfish’ cipher. For more information see the documentation.

Set the cryptmethod to use the blowfish cipher
:setlocal cm=blowfish 

Documentation on Vim Encryption
:h :X

Typically uses swap files that can compromise the security of the encrypted file. You can turn this off by using the ’n’ flag.

vim -xn <filename>

Integrating with GPG

In order to seamlessly integrate with GPG encrypted files you will need to add the following to your .vimrc file

" Transparent editing of gpg encrypted files.
" By Wouter Hanegraaff
augroup encrypted
  au!

  " First make sure nothing is written to ~/.viminfo while editing
  " an encrypted file.
  autocmd BufReadPre,FileReadPre *.gpg set viminfo=
  " We don't want a swap file, as it writes unencrypted data to disk
  autocmd BufReadPre,FileReadPre *.gpg set noswapfile

  " Switch to binary mode to read the encrypted file
  autocmd BufReadPre,FileReadPre *.gpg set bin
  autocmd BufReadPre,FileReadPre *.gpg let ch_save = &ch|set ch=2
  " (If you use tcsh, you may need to alter this line.)
  autocmd BufReadPost,FileReadPost *.gpg '[,']!gpg --decrypt 2> /dev/null

  " Switch to normal mode for editing
  autocmd BufReadPost,FileReadPost *.gpg set nobin
  autocmd BufReadPost,FileReadPost *.gpg let &ch = ch_save|unlet ch_save
  autocmd BufReadPost,FileReadPost *.gpg execute ":doautocmd BufReadPost " . expand("%:r")

  " Convert all text to encrypted text before writing
  " (If you use tcsh, you may need to alter this line.)
  autocmd BufWritePre,FileWritePre *.gpg '[,']!gpg --default-recipient-self -ae 2>/dev/null
  " Undo the encryption so we are back in the normal text, directly
  " after the file has been written.
  autocmd BufWritePost,FileWritePost *.gpg u
augroup END

Source: Vim Wiki—Encryption

This works by detecting the extension on the files you are opening with Vim. This allows you to open, edit, and save files as if they were plain text in a seamless fashion.

Now you can create a new GPG encrypted file or edit an existing GPG encrypted doing this:

vim <filename>.gpg

This should prompt you for GPG password either with a GUI window or command line depending on your environment’s configuration. If the file did not exist Vim creates one and when you save it will encrypt when you write and quit.

Another thing I like to do is save a file that I have already decrypted, but want to save in plain text. This can be done by simply opening the encrypted GPG file as seen above and change the extension when saving. Simply save like so:

:w <newfilename>.txt

Any extension other than .gpg will save your file as plain text.

Reason to use GPG

Much safer encryption as it uses GPG
No swap files thanks to this line in .vimrc autocmd BufReadPre,FileReadPre *.gpg set noswapfile

Of course when using Vim there are many features and many different ways of doing this. This is simply how I use Vim to easily work with encrypted files in my daily life.

For more information on Vim and external encryption programs please see and making use of GPG:

Points of Interest

2012-05-15T00:00:00+00:00

It’s been a fairly straight forward week at work, but I have stumbled a few interesting finds along the way this week.

Vim Adventures

Finally! A game based approach to learning Vim keyboard commands. I was hoping someone would do this. It’s just getting started (only two levels) and sadly, it looks like it’ll be charging money to unlock higher levels. However, some things are worth paying for. I’ve found just playing the first two levels a few times have helped retrain my brain to not take my fingers off the home row. It’s still quite buggy and seems to only work in Chrome. I found several times I needed to close all my Chrome windows after playing. Also, incognito mode seems to help with the bugs, as it disables all extensions you may have installed.

MySQL query comments in Rails

Ever wanted to know where that slow query was being called from? Well, if you’re using MySQL with your Rails 2.3.x or 3.x.x app, you can get debug information about what controller’s action made the call. Check out 37Signals new marginalia gem.

How to use EC2 as a web proxy

Kevin Burke provides a very detailed HOWTO article for working around restrictions you may experience in the course of an Internet based life. Pretty amazing what Amazon’s free usage tier puts out there; of course it’s only free for 12 months.

Include PIDs in your Logs

For many Rails developers we get comfortable looking at development log files. Sometimes when I have to investigate a customer issue on a production server using logs, I wished I had the level of detail the development logger had. While that’s a wish, I’m finding it mandatory to include PID numbers in my production logs. In production systems with multiple requests being handled simultaneously, Rails logs start to become unusable. It’s not clear which log lines are from which requests. Adding a PID in front of the time stamp can help untangle the mess. Here are some example approaches to this for Rails 3.x.x and Rails 2.3.x. Also, if you’re really a log-lover and manage a lot of servers, check out Papertrail, it looks very impressive for $7/mo.

Spectrum Shortages - Why it’s happening and what can be done

Telecom is not an area I have much familiarity with, but I found this article to be an interesting read. For example did you know that that largest owner of spectrum licenses are “under-capitalized or unwilling to build out networks” to use the spectrum? So while AT&T and Verizon struggle to meet the iPhone 4S’s data demands (twice as much as iPhone 4!), “there are some companies that have spectrum, but they’re struggling financially. Or they aren’t quite sure what to do with the spectrum. And others that have the money and business model, but need the spectrum.” It seems the way out of the mess is 4G, offering to improve the efficiency of spectrum use by 700 percent.

Editing large files in place

2009-12-13T00:00:00+00:00

Running out of disk space seems to be an all too common problem lately, especially when dealing with large databases. One situation that came up recently was a client who needed to import a large Postgres dump file into a new database. Unfortunately, they were very low on disk space and the file needed to be modified. Without going into all the reasons, we needed the databases to use template1 as the template database, and not template0. This was a very large, multi-gigabyte file, and the amount of space left on the disk was measured in megabytes. It would have taken too long to copy the file somewhere else to edit it, so I did a low-level edit using the Unix utility dd. The rest of this post gives the details.

To demonstrate the problem and the solution, we’ll need a disk partition that has little-to-no free space available. In Linux, it’s easy enough to create such a thing by using a RAM disk. Most Linux distributions already have these ready to go. We’ll check it out with:

$ ls -l /dev/ram*
brw-rw---- 1 root disk 1,  0 2009-12-14 13:04 /dev/ram0
brw-rw---- 1 root disk 1,  1 2009-12-14 22:27 /dev/ram1

From the above, we see that there are some RAM disks available (there are actually 16 of them available on my box, but I only showed two). Here’s the steps to create a usable partition from /dev/ram1, and to then check the size:

$ mkdir /home/greg/ramtest

$ sudo mke2fs /dev/ram1
mke2fs 1.41.4 (27-Jan-2009)
Filesystem label=
OS type: Linux
Block size=1024 (log=0)
Fragment size=1024 (log=0)
4096 inodes, 16384 blocks
819 blocks (5.00%) reserved for the super user
First data block=1
Maximum filesystem blocks=16777216
2 block groups
8192 blocks per group, 8192 fragments per group
2048 inodes per group
Superblock backups stored on blocks:
        8193

Writing inode tables: done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 29 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.

$ sudo mount /dev/ram1 /home/greg/ramtest

$ sudo chown greg:greg /home/greg/ramtest

$ df -h /dev/ram1
Filesystem            Size  Used Avail Use% Mounted on
/dev/ram1              16M  140K   15M   1% /home/greg/ramtest

First we created a new directory to server as the mount point, then we used the mke2fs utility to create a new file system (ext2) on the RAM disk at /dev/ram1. It’s a fairly verbose program by default, but there is nothing in the output that’s really important for this example. Then we mounted our new filesystem to the directory we just created. Finally, we reset the permissions on the directory such that an ordinary user (e.g. ‘greg’) can read and write to it. At this point, we’ve got a directory/filesystem that is just under 16 MB large (we could have made it much closer to 16 MB by specifying a -m 0 to mke2fs, but the actual size doesn’t matter).

To simulate what happened, let’s create a database dump and then bloat it until there it takes up all available space:

$ cd /home/greg/ramtest

$ pg_dumpall > data.20091215.pg

$ ls -l data.20091215.pg
-rw-r--r-- 1 greg greg 3685 2009-12-15 10:42 data.20091215.pg

$ dd seek=3685 if=/dev/zero of=data.20091215.pg bs=1024 count=99999
dd: writing 'data.20091215.pg': No space left on device
13897+0 records in
13896+0 records out
14229504 bytes (14 MB) copied, 0.0814188 s, 175 MB/s

$ df -h .
Filesystem            Size  Used Avail Use% Mounted on
/dev/ram1              16M   15M     0 100% /home/greg/ramtest

First we created the dump, then we found the size of it, and told dd via the ‘seek’ argument to start adding data to it at the 3685 byte mark (in other words, we appended to the file). We used the special file /dev/zero as the ‘if’ (input file), and our existing dump as the ‘of’ (output file). Finally, we told it to chunk the inserts into 1024 bytes at a time, and to attempt to add 999,999 of those chunks. Since this is approximately 100MB, we ran out of disk space quickly, as we intended. The filesystem is now at 100% usage, and will refuse any further writes to it.

To recap, we need to change the first three instances of template0 with template1. Let’s use grep to view the lines:

$ grep --text --max-count=3 template data.20091215.pg
CREATE DATABASE greg WITH TEMPLATE = template0 OWNER = greg ENCODING = 'UTF8';
CREATE DATABASE rand WITH TEMPLATE = template0 OWNER = greg ENCODING = 'UTF8';
CREATE DATABASE sales WITH TEMPLATE = template0 OWNER = greg ENCODING = 'UTF8';

We need the –text argument here because grep correctly surmises that we’ve changed the file from text-based to binary with the addition of all those zeroes on the end. We also used the handy –max-count argument to stop processing once we’ve found the lines we want. Very handy argument when the actual file is gigabytes in size!

There are two major problems with using a normal text editor to change the file. First, the file (in the real situation, not this example!) was very, very large. We only needed to edit something at the very top of the file, so loading the entire thing into an editor is very inefficient. Second, editors need to save their changes somewhere, and there just was not enough room to do so.

Attempting to edit with emacs gives us: emacs: IO error writing /home/greg/ramtest/data.20091215.pg: No space left on device

An attempt with vi gives us: vi: Write error in swap file on startup. “data.20091215.pg” E514: write error (file system full?)

Although emacs gives the better error message (why is vim making a guess and outputting some weird E514 error?), the advantage always goes to vi in cases like this as emacs has a major bug in that it cannot even open very large files.

What about something more low-level like sed? Unfortunately, while sed is more efficient than emacs or vim, it still needs to read the old file and write the new one. We can’t do that writing as we have no disk space! More importantly, in sed there is no way (that I could find anyway) to tell it stop processing after a certain number of matches.

What we need is something really low-level. The utility dd comes to the rescue again. We can use dd to truly edit the file in place. Basically, we’re going to overwrite some of the bytes on disk, without needing to change anything else. First though, we have to figure out exactly which bytes to change. The grep program has a nice option called –byte-offset that can help us out:

$ grep --text --byte-offset --max-count=3 template data.20091215.pg
301:CREATE DATABASE greg WITH TEMPLATE = template0 OWNER = greg ENCODING = 'UTF8';
380:CREATE DATABASE rand WITH TEMPLATE = template0 OWNER = greg ENCODING = 'UTF8';
459:CREATE DATABASE sales WITH TEMPLATE = template0 OWNER = greg ENCODING = 'UTF8';

This tells us the offset for each line, but we want to replace the number ‘0’ in ‘template0’ with the number ‘1’. Rather than count it out manually, let’s just use another Unix utility, hexdump, to help us find the number:

$ grep --text --byte-offset --max-count=3 template data.20091215.pg | hexdump -C
00000000  33 30 31 3a 43 52 45 41  54 45 20 44 41 54 41 42  |301:CREATE DATAB|
00000010  41 53 45 20 67 72 65 67  20 57 49 54 48 20 54 45  |ASE greg WITH TE|
00000020  4d 50 4c 41 54 45 20 3d  20 74 65 6d 70 6c 61 74  |MPLATE = templat|
00000030  65 30 20 4f 57 4e 45 52  20 3d 20 67 72 65 67 20  |e0 OWNER = greg |
00000040  45 4e 43 4f 44 49 4e 47  20 3d 20 27 55 54 46 38  |ENCODING = 'UTF8|
...

Each line is 16 characters, so the first three lines comes to 48 characters, then we add two for the ‘e0’, subtract four for the ‘301:’, and get 301+48+2-4=347. We subtract one more as we want to seek to the point just before that character, and we can now use our dd command:

$ echo 1 | dd of=data.20091215.pg seek=346 bs=1 count=1 conv=notrunc
1+0 records in
1+0 records out
1 byte (1 B) copied, 0.00012425 s, 8.0 kB/s

Instead of an input file (the ‘if’ argument), we simply pass the number ‘1’ via stdin to the dd command. We use our calculated seek, tell it to copy a single byte (bs=1), one time (count=1), and (this is very important!) tell dd NOT to truncate the file when it is done (conv=notrunc). Technically, we are sending two characters to the dd program, the number one and a newline, but the bs=1 argument ensures only the first character is being copied. We can now verify that the change was made as we expected:

$ grep --text --byte-offset --max-count=3 TEMPLATE data.20091215.pg
301:CREATE DATABASE greg WITH TEMPLATE = template1 OWNER = greg ENCODING = 'UTF8';
380:CREATE DATABASE rand WITH TEMPLATE = template0 OWNER = greg ENCODING = 'UTF8';
459:CREATE DATABASE sales WITH TEMPLATE = template0 OWNER = greg ENCODING = 'UTF8';

Now for the other two entries. From before, the magic number is 45, so we now add 380 to 45 to get 425. For the third line, the name of the database is 1 character longer so we add 459+45+1 = 505:

$ echo 1 | dd of=data.20091215.pg seek=425 bs=1 count=1 conv=notrunc
1+0 records in
1+0 records out
1 byte (1 B) copied, 0.000109234 s, 9.2 kB/s

$ echo 1 | dd of=data.20091215.pg seek=505 bs=1 count=1 conv=notrunc
1+0 records in
1+0 records out
1 byte (1 B) copied, 0.000109932 s, 9.1 kB/s

$ grep --text --byte-offset --max-count=3 TEMPLATE data.20091215.pg
301:CREATE DATABASE greg WITH TEMPLATE = template1 OWNER = greg ENCODING = 'UTF8';
380:CREATE DATABASE rand WITH TEMPLATE = template1 OWNER = greg ENCODING = 'UTF8';
459:CREATE DATABASE sales WITH TEMPLATE = template1 OWNER = greg ENCODING = 'UTF8';

Success! On the real system, the database was loaded with no errors, and the large file was removed. If you’ve been following along and need to cleanup:

$ cd ~
$ sudo umount /home/greg/ramtest
$ rmdir ramtest

Keep in mind that dd is a very powerful and thus very dangerous utility, so treat it with care. It can be invaluable for times like this however!

Vim tip of the day: tabbed editing

2009-03-11T00:00:00+00:00

How many tabs does your browser have open? I have 17 tabs open in Firefox presently (and opened / closed about 12 while writing this post). Most users will agree that tabs have changed the way they use the Web. Even IE, which has spawned a collection of shells for tabbed browsing, now supports it natively. Tabs allow for a great saving in screen real-estate, and in many cases, better interaction among the various open documents. Considering how much time programmers spend in their text editors, it therefore seems logical that the editor should provide the same functionality.

And the Vim developers agree. Although Vim calls them “tab-pages”, the functionality is there, waiting to be used. Before reading any further, ensure that your Vim supports tabs. You can do this by running this command, on anything resembling Unix:

vim --version | fgrep +windows

If you don’t see any output, check your vendor’s packaging system for something like vim-full. If you don’t have a Vim available with the windows feature, go get one and come back.

Now that you can use tabs, let’s get started. One way to open tabs is via the command line. Vim uses the -p option to determine how many tab-pages to open. This functions like the -[oO] options for windows in that it accepts a numeric argument, but defaults to one for each specified file.

Let’s propose a hypothetical situation: I want to compare the implementation of has() in both Moose and Mouse. Therefore, it might be convenient to have the two files open in two tab-pages in Vim. Presuming I’m in the same directory as both files, I would open Vim with a tab-page for each file like so:

vim -p Moose.pm Mouse.pm

There will be a bar at the top of the terminal: That’s the tab bar. It has every open tab, plus an X at the far right. That’s great, and all, but how does one use this? There are a number of ways, depending upon your configuration. First, there’s the basic tab page commands. These are:

gt Advance to the next rightmost tab, cycling back to the first.
gT Advance to the next leftmost tab, cycling back to the last.
{count}gt Go to the {count} tab.

These commands work in both command-line Vim (for those of you working on remote machines) and in gvim (for those of you writing new code). Additionally, gvim and Vims that recognize mouse input on a terminal (:help mouse-using) recognize clicks on the tabs or the X in the upper right.

This tip has touched on the very basics of tab-pages, but the information covered here is enough to be useful (and is all I use on a regular basis). However, the curious should definitely read the Vim reference manual for more info, with :help tabpage.

Vim Tip of the Day: running external commands in a shell

2009-03-10T00:00:00+00:00

A common sequence of events when editing files is to make a change and then need to test by executing the file you edited in a shell. If you’re using Vim, you could suspend your session (ctrl-Z), and then run the command in your shell.

That’s a lot of keystrokes, though.

So, instead, you can use Vim’s built-in “run a shell command”!

:!{cmd} Run a shell command, shows you the output and prompts you before returning to your current buffer.

Even sweeter, is to use the Vim special character for current filename: %

Here’s :! % in action!

A few more helpful shortcuts related to executing things in the shell:

:! By itself, runs the last external command (from your shell history)
:!! Repeats the last command
:silent !{cmd} Eliminates the need to hit enter after the command is done
:r !{cmd} Puts the output of $cmd into the current buffer.