• Home

  • Custom Ecommerce
  • Application Development
  • Database Consulting
  • Cloud Hosting
  • Systems Integration
  • Legacy Business Systems
  • Security & Compliance
  • GIS

  • Expertise

  • About Us
  • Our Team
  • Clients
  • Blog
  • Careers

  • VisionPort

  • Contact
  • Our Blog

    Ongoing observations by End Point Dev people

    Aligning monospace font text columns with an old Unix tool

    Jon Jensen

    By Jon Jensen
    May 30, 2022

    Photo of old wooden bridge with moss on it Photo by Garrett Skinner

    A blast from 1990: column

    A while back I learned of a nice old Unix command-line tool called column. It first appeared in 4.3BSD-Reno, released in July 1990. (This is not to be confused with the different, even older Unix tool col.)

    column formats plain text into nice columns based on the width of the input separated by tabs or groups of spaces.

    For example, take this mess found in a server’s /etc/fstab file defining filesystem mount points. It is a real example lightly redacted to remove business details. You may need to scroll right to see the end of the fairly long lines:

    /data3/customer_uploads   /home/interch/htdocs/shared/customer_uploads none    rw,bind 0   0
    /data3/customer_images    /home/interch/htdocs/shared/customer_images  none    rw,bind 0   0
    /data3/images/items       home/interch/htdocs/images/items     none    rw,bind 0       0
    /data3/images/thumb       home/interch/htdocs/images/thumb     none    rw,bind 0       0
    /data3/upload_images       /home/interch/upload_images     none    rw,bind 0       0
    /data3/design    /home/interch/htdocs/shared/design  none    rw,bind 0   0
    /data3/design_temp   /home/interch/htdocs/shared/design_temp none    rw,bind 0   0
    /data3/reports/cat1       /home/interch/htdocs/cat-numero-uno/images/reports     none    rw,bind 0       0
    /data3/reports/cat2       /home/interch/htdocs/cat-zahl-zwei/images/reports     none    rw,bind 0       0
    /data3/reports/cat3	/home/interch/htdocs/cat-number-three/images/reports	none	rw,bind	0	0
    /data3/reports/cat4    /home/interch/htdocs/cat-quatre/images/reports     none    rw,bind 0       0
    /data3/reports/cat5    /home/interch/htdocs/cat-pět/images/reports     none    rw,bind 0       0
    /data3/shared_var /home/interch/catalogs/shared/var none    rw,bind 0 0
    

    That is really unsightly and includes a mix of spaces and tabs.

    If we feed it to column -t we get the same data aligned much nicer:

    % column -t /etc/fstab
    /data3/customer_uploads  /home/interch/htdocs/shared/customer_uploads          none  rw,bind  0  0
    /data3/customer_images   /home/interch/htdocs/shared/customer_images           none  rw,bind  0  0
    /data3/images/items      home/interch/htdocs/images/items                      none  rw,bind  0  0
    /data3/images/thumb      home/interch/htdocs/images/thumb                      none  rw,bind  0  0
    /data3/upload_images     /home/interch/upload_images                           none  rw,bind  0  0
    /data3/design            /home/interch/htdocs/shared/design                    none  rw,bind  0  0
    /data3/design_temp       /home/interch/htdocs/shared/design_temp               none  rw,bind  0  0
    /data3/reports/cat1      /home/interch/htdocs/cat-numero-uno/images/reports    none  rw,bind  0  0
    /data3/reports/cat2      /home/interch/htdocs/cat-zahl-zwei/images/reports     none  rw,bind  0  0
    /data3/reports/cat3      /home/interch/htdocs/cat-number-three/images/reports  none  rw,bind  0  0
    /data3/reports/cat4      /home/interch/htdocs/cat-quatre/images/reports        none  rw,bind  0  0
    /data3/reports/cat5      /home/interch/htdocs/cat-pět/images/reports           none  rw,bind  0  0
    /data3/shared_var        /home/interch/catalogs/shared/var                     none  rw,bind  0  0
    

    That isn’t just prettier! It also makes some things stand out prominently at a glance:

    • In the second column there are two mount points not starting with /.
    • It’s easy to see that most of the paths in the second column start with /home/interch/htdocs/ and the few that don’t, stand out.
    • The final 4 columns are all identical, which was unclear in the unaligned original.

    Text tables to JSON

    The util-linux version of column includes extra options beyond the original. One very useful one is -J or --json which produces a JSON object for each row based on column names you define in the argument -N or --table-columns.

    Reviewing man fstab for details on what each column is used for in the sample above, we can instruct column to produce JSON output like this:

    % column -J -n fstab -N spec,file,vfstype,mntops,freq,passno /etc/fstab
    

    Which gives us this result:

    {
       "fstab": [
          {
             "spec": "/data3/customer_uploads",
             "file": "/home/interch/htdocs/shared/customer_uploads",
             "vfstype": "none",
             "mntops": "rw,bind",
             "freq": "0",
             "passno": "0"
          },{
             "spec": "/data3/customer_images",
             "file": "/home/interch/htdocs/shared/customer_images",
             "vfstype": "none",
             "mntops": "rw,bind",
             "freq": "0",
             "passno": "0"
          },{
             "spec": "/data3/images/items",
             "file": "home/interch/htdocs/images/items",
             "vfstype": "none",
             "mntops": "rw,bind",
             "freq": "0",
             "passno": "0"
          },{
             "spec": "/data3/images/thumb",
             "file": "home/interch/htdocs/images/thumb",
             "vfstype": "none",
             "mntops": "rw,bind",
             "freq": "0",
             "passno": "0"
          },{
             "spec": "/data3/upload_images",
             "file": "/home/interch/upload_images",
             "vfstype": "none",
             "mntops": "rw,bind",
             "freq": "0",
             "passno": "0"
          },{
             "spec": "/data3/design",
             "file": "/home/interch/htdocs/shared/design",
             "vfstype": "none",
             "mntops": "rw,bind",
             "freq": "0",
             "passno": "0"
          },{
             "spec": "/data3/design_temp",
             "file": "/home/interch/htdocs/shared/design_temp",
             "vfstype": "none",
             "mntops": "rw,bind",
             "freq": "0",
             "passno": "0"
          },{
             "spec": "/data3/reports/cat1",
             "file": "/home/interch/htdocs/cat-numero-uno/images/reports",
             "vfstype": "none",
             "mntops": "rw,bind",
             "freq": "0",
             "passno": "0"
          },{
             "spec": "/data3/reports/cat2",
             "file": "/home/interch/htdocs/cat-zahl-zwei/images/reports",
             "vfstype": "none",
             "mntops": "rw,bind",
             "freq": "0",
             "passno": "0"
          },{
             "spec": "/data3/reports/cat3",
             "file": "/home/interch/htdocs/cat-number-three/images/reports",
             "vfstype": "none",
             "mntops": "rw,bind",
             "freq": "0",
             "passno": "0"
          },{
             "spec": "/data3/reports/cat4",
             "file": "/home/interch/htdocs/cat-quatre/images/reports",
             "vfstype": "none",
             "mntops": "rw,bind",
             "freq": "0",
             "passno": "0"
          },{
             "spec": "/data3/reports/cat5",
             "file": "/home/interch/htdocs/cat-pět/images/reports",
             "vfstype": "none",
             "mntops": "rw,bind",
             "freq": "0",
             "passno": "0"
          },{
             "spec": "/data3/shared_var",
             "file": "/home/interch/catalogs/shared/var",
             "vfstype": "none",
             "mntops": "rw,bind",
             "freq": "0",
             "passno": "0"
          }
       ]
    }
    

    JSON takes a lot more room, and of course is not the format Linux expects for this particular file, but transforming tabular data to JSON format in other situations can be more readable for exchange across different systems since each field is labeled, and unused fields can be omitted. Plus JSON syntax is rigorously defined, nested data structures are possible, etc.

    Columnizing lists

    All versions of column can also columnize lists, either horizontally (across) or vertically (down). Take this list of people’s names:

    Amy
    Anna
    Bob
    Brenda
    Cameron
    Doug
    Emily
    Frank
    Jane
    Jill
    Jim
    Joe
    John
    Karen
    Kate
    Liz
    Mary
    Mike
    Sarah
    Steve
    Victoria
    

    Put that in file /tmp/names and column will format it in columns fitting the width of your terminal:

    % column /tmp/names
    Amy		Cameron		Jane		John		Mary		Victoria
    Anna		Doug		Jill		Karen		Mike
    Bob		Emily		Jim		Kate		Sarah
    Brenda		Frank		Joe		Liz		Steve
    

    It uses 2 or more tab characters to separate the columns, based on standard terminal 8-space tab stops, so the above doesn’t look right here on the web.

    What appears in my terminal looks like:

    % column /tmp/names
    Amy             Cameron         Jane            John            Mary            Victoria
    Anna            Doug            Jill            Karen           Mike
    Bob             Emily           Jim             Kate            Sarah
    Brenda          Frank           Joe             Liz             Steve
    

    Or you can use column -t that we discussed earlier to format the columns more compactly with spaces:

    % column /tmp/names | column -t
    Amy     Cameron  Jane  John   Mary   Victoria
    Anna    Doug     Jill  Karen  Mike   
    Bob     Emily    Jim   Kate   Sarah  
    Brenda  Frank    Joe   Liz    Steve  
    

    You can also ask for the list to be delivered horizontally, rather than vertically, with column -x:

    % column -x /tmp/names | column -t
    Amy    Anna   Bob       Brenda  Cameron  Doug
    Emily  Frank  Jane      Jill    Jim      Joe
    John   Karen  Kate      Liz     Mary     Mike
    Sarah  Steve  Victoria                   
    

    Note that it isn’t doing anything to affect your ordering. You can order the lines in your original file however you want and it will preserve them. But other tools can help you here: Use sort -u to sort alphabetically and remove duplicates.

    For more options and details see the column man page for the util-linux column version or FreeBSD column version (same as macOS).

    A blast from 1974: pr

    There is an even older Unix tool for columnizing lists in the same way. It is called pr and dates to First Edition Unix in 1971, but did not gain the options we are using here until Fifth Edition Unix in 1974 as seen in the V6 pr man page.

    We need to tell it how many columns to produce, so we will ask for 6 columns as column was doing above. Note that pr emits a curious mix of tabs and spaces, which cat -t reveals here as ^I (since a tab is the same thing as Control+I):

    % pr -t -6 /tmp/names | cat -t
    Amy^I    Cameron^IJane^I    John^ILiz^I    Sarah
    Anna^I    Doug^IJill^I    Karen^IMary^I    Steve
    Bob^I    Emily^IJim^I    Kate^IMike^I    Victoria
    Brenda^I    Frank^IJoe
    

    But in a terminal it looks fine:

    % pr -t -6 /tmp/names
    Amy         Cameron     Jane        John        Liz         Sarah
    Anna        Doug        Jill        Karen       Mary        Steve
    Bob         Emily       Jim         Kate        Mike        Victoria
    Brenda      Frank       Joe
    

    See the many more options of pr in the GNU coreutils pr man page and FreeBSD pr man page (same as macOS).

    A blast from 1979: expand

    A useful tool for dealing with tabs is expand, which first appeared in 3BSD in 1979. (Despite the FreeBSD and macOS man pages saying it appeared in 1BSD, I don’t see it there or in 2BSD.)

    We can use it to convert tabs to spaces just like a terminal would:

    % pr -t -6 /tmp/names | expand | cat -t    
    Amy         Cameron     Jane        John        Liz         Sarah
    Anna        Doug        Jill        Karen       Mary        Steve
    Bob         Emily       Jim         Kate        Mike        Victoria
    Brenda      Frank       Joe
    

    expand accepts the optional -t argument with a number to use as tabstop width instead of the default 8.

    And unexpand

    I have used expand for years, but only recently learned of unexpand which goes the other way, converting runs of spaces into tabs:

    % pr -t -6 /tmp/names | expand | unexpand -a         
    Amy         Cameron     Jane        John        Liz         Sarah
    Anna        Doug        Jill        Karen       Mary        Steve
    Bob         Emily       Jim         Kate        Mike        Victoria
    Brenda      Frank       Joe
    

    It looks fine in the terminal, but let’s see if it actually used tabs:

    % pr -t -6 /tmp/names | expand | unexpand -a | cat -t
    Amy^I    Cameron^IJane^I    John^ILiz^I    Sarah
    Anna^I    Doug^IJill^I    Karen^IMary^I    Steve
    Bob^I    Emily^IJim^I    Kate^IMike^I    Victoria
    Brenda^I    Frank^IJoe
    

    Interesting… For some reason unexpand doesn’t actually convert all the spaces to tabs, but just one initial tab per gutter between columns. Running the output through unexpand -a again has no further effect. Strange.

    The GNU coreutils version of unexpand that lives in most Linux systems is what I would call bug-compatible with the BSD version in this regard.

    Oh, well. There’s probably a reason.

    Since it has handled turning the initial tabstop’s varying number of spaces into a tab, we can easily remove the remaining fixed multiples of spaces on our own for a more compact list:

    % pr -t -6 /tmp/names | expand | unexpand -a | sed 's/    //g'         
    Amy     Cameron Jane    John    Liz     Sarah
    Anna    Doug    Jill    Karen   Mary    Steve
    Bob     Emily   Jim     Kate    Mike    Victoria
    Brenda  Frank   Joe
    

    Ready at hand

    column, pr, expand, and unexpand all come with most Linux and BSD systems, including macOS. It is amazing what great old tools are on many of our computers all the time, waiting to be used!

    You can use these programs as filters inside your favorite text editor or IDE. For example, to achieve the same list columnization as above you can select a block of text and send it to external commands:

    • In Vim, visually select text with v or V and then type !column | expand and press Enter.
    • In VS Code you can install an extension called Filter Text (by yhirose). Once installed, type Control+K Control+F (⌘K ⌘F on macOS) and then column | expand and Enter.
    • In IntelliJ IDEA you can install a plugin called Shell Filter (by Dennis Plöger). Once installed, select your text, then choose menu item Edit > Custom Shell Filter and then type column -c 80 | expand and Enter.
    • Most other editors have a way to do this too. Search for “filter”, “pipe”, and/or “command”.

    Your selection will be replaced by the output.

    Unicode

    These tools come from an era when one byte of input resulted in one visual character, so for any UTF-8 characters outside the limited classic ASCII character set, we could expect old tools may miscalculate the needed space between columns.

    Originally in my testing they appeared not to be aware of some Unicode characters’ width, perhaps due to combining characters or alternate forms. But surprisingly they have been modernized and seem to work fine with most Unicode characters such as Latin letters with diacritics, letters in other alphabets, Chinese characters, and even emoji! 😊

    tips tools vim vscode intellij-idea


    Comments