Bash expansion techniques for a more efficient workflow

2025-07-10T00:00:00+00:00

For any project, you need a quick and efficient way to wrangle your files. If you use Unix, Bash and Zsh are powerful tools to help achieve this.

I recently needed to rename a file so that all its underscores were replaced with dash characters, to match the convention of the project. I could do this manually pretty quickly, but I knew there was a bash built-in one-liner waiting to be discovered, so I went down the rabbit hole to learn about Bash’s shell expansions and history expansion. See the “history expansion” section for how I solved the underscore/dash issue.

Bash has seven types of expansion:

brace expansion
tilde expansion
parameter and variable expansion
command substitution
arithmetic expansion
word splitting
filename expansion

The documentation is good and concise for each of these, so rather than try to recreate it, I’ll go over examples of how I use some of them.

Shell parameter expansion

Example: batch converting images to WebP

I use parameter expansion frequently while maintaining this blog. We serve images in WebP format, so I generally loop over all the JPEGs and/or PNGs (after cropping and/or scaling) and convert them using cwebp:

for f in *.png; do
  cwebp -q 80 $f -o ${f%.png}.webp
done

This makes use of parameter expansion (${}) and the % character, which deletes the shortest occurrence of the word it precedes (png) from the end of the parameter ($f, the filename). Then, you can insert a string immediately after the expansion and Bash will concatenate them together, making a new output filename.

If you use %% instead, it deletes the longest occurrence of the word from the end, instead of the shortest. This would only apply if you are doing pattern matching.

You can use the # character to remove a word from the beginning of the string, conversely to how % removes a match from the end.

The formal definition of word in the Bash manual is “A sequence of characters treated as a unit by the shell. Words may not include unquoted metacharacters.”

I looked this up so I would know the distinction between words and parameters. Parameters are more specific: mainly just variables, positional parameters, and a few special parameters.

Alternate method with parameter expansion search and replace

As a bonus, my co-worker Josh showed me how he uses search and replace to solve the same problem. Note that this is parameter expansion search and replace, which has different syntax from history expansion search and replace (shown two examples down). Also, they both search for a pattern, not a regular expression, so . will match the literal . character, etc.

for f in *.png; do
  cwebp -q 80 $f -o ${f/png/webp}
done

I use that version much more often than the %, as it (at least to me) makes it more clear what’s happening when doing a string replacement. Also, it doesn’t do anything if png isn’t found, whereas the above % method always appends .webp… Which may be better anyway, depending on what you want to happen in that case. :)

Example: counting file endings

I haven’t used this one as often, but ##pattern removes the longest match of the word from the beginning of the parameter. I cooked up an example to count the occurrences of each file ending in the current directory:

for f in *.*; do
  echo ${f##*.}
done | sort | uniq -c

It’s not especially robust (e.g., it would show .tar.gz as simply .gz), but this is an example of a lightweight and versatile script with just Bash builtins and standard commands. Knowing the tools you’re working with allows you to adapt them to your current situation.

https://www.gnu.org/software/bash/manual/html_node/Shell-Parameter-Expansion.html

History Expansion

Another type of expansion I use frequently is history expansion. The history expansion character is !, and there are several useful ways to use history within Bash commands.

!! repeats the most recent command. This is especially useful if you need to rerun a command with sudo. I often find myself running sudo !! after accidentally trying to install a program without superuser privileges.

You can also expand any argument from the previous command using !:n, where n is the argument from the previous command, numbered from 0. More often than this, I use the shorthand !$ which references the last argument from the previous command.

For example, if you’ve just run diff really\ long\ filename\ I\ really\ don\'t\ want\ to\ type.txt other.txt, you can edit the first file by typing vim !:1, without having to retype the long filename.

You can also reference previous words in the current command using the !# event designator:

$ echo words and more !#1
words and more words

Example: Replacing underscores with dashes in a filename

Here’s how I solved the underscore/dash issue, as promised. I used the s modifier to search and replace within the backreferenced word within the current command:

$ mv synthesized_beef_menu.md !#:1:gs/_/-
mv synthesized_beef_menu.md synthesized-beef-menu.md

Note that this is real output; Bash prints the expanded command before showing the output.

So simple! We just reference the second word (index 1) in the current command and globally subsitute - for _.

Notice that history expansion works differently from the ${} parameter expansion notation we saw previously. Rather than using the delimiter as a command, in history expansion you can use one or more modifiers separatd by : characters. If you have two files named green-old.txt and green-new (no file ending), you can make the second out of the first:

$ cat green-old.txt
...
$ cat !:1:r:s/old/new
cat green-new
...

This references word 1 from the previous command, removes the filename extension, leaving the root filename, then replaces “old” with “new”. Not terribly useful for a single file, but this type of expansion could easily be used in a quick script to move a large number of files into a new format.

To replace globally (not just once per line), you have to put the modifier before the search/replace pattern: !!:0:gs/pattern/replacement. This is different from sed-style search and replace flags, which go at the end after a terminal slash.

https://www.gnu.org/software/bash/manual/html_node/History-Interaction.html

Other types of expansion

I recommend looking through the documentation for expansions and applying it to your own routines. A little goes a long way! If you get in the habit of noticing inefficiency and fixing it with the shell, you’ll find yourself finishing menial tasks quicker, leaving more time to spend solving problems that matter.

How to Analyze Application Logs and Extract Actionable Insights

2025-02-28T00:00:00+00:00

Photo by Timo C. Dinger on Unsplash

Logs often accumulate unnoticed—until something breaks. Then, they suddenly become vital tools for diagnosing issues. By combining just a few command-line techniques, you can quickly spot recurring problems, identify suspicious activity, and strengthen your application’s defenses.

Below is a sample error log we’ll reference in the examples:

[2025-01-01 12:34:56] [client 192.168.1.1:12345] PHP Notice: Undefined variable $foo in /var/www/html/index.php on line 45 | referer: http://example.com/index.php
[2025-01-01 12:34:56] [client 192.168.1.1:12345] PHP Notice: Undefined variable $foo in /var/www/html/index.php on line 45 | referer: http://example.com/index.php
[2025-01-01 12:34:57] [client 192.168.1.1:12345] PHP Notice: Undefined variable $foo in /var/www/html/index.php on line 45 | referer: http://example.com/index.php
[2025-01-01 12:35:00] [client 10.0.0.2:6789] PHP Warning: Division by zero in /var/www/html/script.php on line 23 | referer: http://example.com/script.php
[2025-01-01 13:35:01] [client 10.0.0.2:6789] PHP Warning: Division by zero in /var/www/html/script.php on line 23 | referer: http://example.com/script.php
[2025-01-01 13:36:10] [client 192.168.1.1:12345] PHP Fatal error: Call to undefined function baz() in /var/www/html/lib.php on line 78 | referer: http://example.com/index.php
[2025-01-01 14:36:11] [client 172.16.0.5:54321] PHP Fatal error: Call to undefined function baz() in /var/www/html/lib.php on line 78 | referer: http://example.com/index.php

A Note on Commands

Throughout this blog, you’ll notice repeated use of commands like awk, grep, sed, uniq, and sort. These tools are indispensable for log analysis, allowing you to filter, transform, and summarize data efficiently. Here are some tips for learning these tools:

Use man <command> (e.g., man awk) to access the manual and dive deeper into a specific command
Experiment with small examples to understand how commands like awk process patterns or how grep extracts data
For a broader look at how these commands enhance productivity, check out my blog post: Practical Linux Command Line Tips for Productivity and Efficiency

These commands are not just for log analysis—they’re powerful for any data manipulation task you encounter. With practice, they can become essential to your workflow.

1. Debugging Application Errors

1.1 Summarize Frequent Issues

sed -E 's/^\[.*\] //' error.log | sort | uniq -c | sort -nr

Purpose: Remove timestamps/client’s IP, then sort and count duplicates
Outcome: A quick snapshot of which errors appear most often

3 PHP Notice: Undefined variable $foo ...
2 PHP Warning: Division by zero ...
2 PHP Fatal error: Call to undefined function baz() ...

1.2 Categorize Errors by Type

grep -oE 'PHP [^:]+' error.log | sort | uniq -c | sort -nr

Purpose: Extract error types (e.g., Notice, Warning, Fatal error)
Outcome: Know whether notices, warnings, or fatal errors dominate

3 Notice
2 Warning
2 Fatal error

1.3 Find Problematic Scripts

awk -F'in ' '{print $2}' error.log | awk '{print $1}' | sort | uniq -c | sort -nr

Purpose: Identify which files generate the most errors
Outcome: Pinpoint error hotspots for focused debugging

3 /var/www/html/index.php
2 /var/www/html/script.php
2 /var/www/html/lib.php

2. Monitoring System Behavior

2.1 Track Problematic IPs

awk -F'\[client ' '{print $2}' error.log | awk -F':' '{print $1}' | sort | uniq -c | sort -nr

Purpose: Count how many errors each IP triggers
Outcome: Identify suspicious or high-traffic IPs

4 192.168.1.1
2 10.0.0.2
1 172.16.0.5

2.2 Spot High-Error Time Periods

awk -F'[][]' '{split($2, time, ":"); print $1, time[1]":"time[2]":00"}' error.log | sort | uniq -c | sort -nr

Purpose: Group errors by minute, revealing spikes or trends
Outcome: Correlate error surges with deployments or traffic peaks

3 2025-01-01 12:34:00
1 2025-01-01 14:36:00
1 2025-01-01 13:36:00
...

2.3 Analyze IP–Referer Pairs

grep "referer:" error.log | awk -F'\\[client ' '{print $2}' | awk -F':| referer: ' '{print $1, $3}' | sort | uniq -c | sort -nr

Purpose: Match IP addresses with the URLs they refer from
Outcome: Detect repeat offenders or malicious traffic patterns

3 192.168.1.1 Undefined variable $foo ...
2 10.0.0.2 Division by zero ...
1 172.16.0.5 Call to undefined function baz() ...

3. Enhancing Security

3.1 Watch Sensitive URLs

grep -E "admin|login" script_error_frequency.txt | sort -nr

Purpose: Flag frequent hits on admin or login pages
Outcome: Gauge whether sensitive endpoints are under attack

2 /var/www/html/login.php
2 /var/www/html/admin.php

3.2 Pinpoint Critical Times

grep -E "admin|login" error.log | awk -F'[][]' '{split($2, time, ":"); print $1, time[1]":"time[2]":00"}' | sort | uniq -c | sort -nr

Purpose: Identify specific time windows for sensitive access
Outcome: Cross-reference suspicious activity with your security logs

2 2025-01-01 12:34:00
1 2025-01-01 13:35:00
1 2025-01-01 12:35:00

Conclusion

These commands demonstrate how quickly you can glean insights from logs. With a little creativity, you can expand them to track response times, detect performance bottlenecks, and safeguard critical endpoints. Whether you’re tackling PHP errors or any other type of log data, the same principles apply: filter, sort, count, and investigate.

Curious to learn more? Combine these strategies with automation tools, integrate them into CI/CD pipelines, or hook them up to visual dashboards. Your logs will become a gold mine of actionable information.

Practical Linux Command Line Tips for Productivity and Efficiency

2024-06-22T00:00:00+00:00

Feeling stuck with basic Linux commands? You’re not alone! Many people know some commands but don’t feel efficient. The good news is that with some know-how and common commands, you can transform your skills.

In this post, you’ll learn to combine commands with pipes, master shell techniques, efficiently recall and edit past commands, and navigate the filesystem with speed. Let’s dive in!

1. Combining Commands with Pipes

Understanding How Pipes Work

Ever wish your commands could work together? Pipes make it happen! They let one command pass its output to another, creating a smooth workflow.

Commands take input (stdin) and give output (stdout). The pipe symbol (|) links them, making life easier.

For example, ls -l /bin lists files in the /bin directory, but the amount of output is overwhelming. Try ls -l /bin | less to view one screen at a time—like having a personal organizer! This command pipes the output of the ls command into the pager less, which is very useful for reading large amounts of data.

Let’s dive into pipes and practice them with commands like wc, head, cut, grep, sort, and uniq to handle data like a pro. If any of these commands are new to you, use man (short for “manual”) followed by the command name to learn more. For example:

man grep

Practical Examples: Real-World Use Cases for Combining Commands

Example 1: Extracting Information from Log Files

Task: Find all error messages in a log file and count them.

Sample Data (application.log):

INFO 2024-06-07 User login successful
ERROR 2024-06-07 Failed to connect to database
INFO 2024-06-07 User logout
ERROR 2024-06-07 Timeout while retrieving data
INFO 2024-06-07 Session expired
ERROR 2024-06-07 Null pointer exception

Command:

grep 'ERROR' application.log | wc -l

Explanation:

grep 'ERROR' application.log searches for lines containing “ERROR” in application.log.
The output is then passed to wc -l, which counts the number of lines, giving you the total number of error messages. wc is short for “word count”, while -l stands for lines.

Expected Output:

Example 2: Processing XML Files for Public Health Data

Task: Extract all patient IDs from multiple XML files and sort them uniquely.

Sample Data (file1.xml, file2.xml):

file1.xml:

<patients>
    <patient>
        <patientId>123</patientId>
        <name>John Doe</name>
    </patient>
    <patient>
        <patientId>124</patientId>
        <name>Jane Smith</name>
    </patient>
</patients>

file2.xml:

<patients>
    <patient>
        <patientId>124</patientId>
        <name>Jane Smith</name>
    </patient>
    <patient>
        <patientId>125</patientId>
        <name>Mary Johnson</name>
    </patient>
</patients>

Command:

grep '<patientId>' *.xml | cut -d '>' -f 2 | cut -d '<' -f 1 | sort | uniq

Explanation:

grep '<patientId>' *.xml searches for lines containing <patientId> in all XML files.
The output is then piped to cut -d'>' -f2 to extract the content after the > character.
cut -d'<' -f1 further extracts the content before the < character.
sort sorts the IDs alphabetically.
uniq removes duplicate entries.

Expected Output:

123
124
125

Example 3: Organizing Text Data into a CSV File

Task: Convert a space-separated file to a CSV file.

Sample Data (data.txt):

Name Age Location
Alice 30 New_York
Bob 25 Los_Angeles
Charlie 35 Chicago

Command:

cat data.txt | tr ' ' ',' > data.csv

Explanation:

cat data.txt reads the content of data.txt.
The output is then piped to tr ' ' ',', which translates spaces to commas.
> data.csv redirects the final output to data.csv. We’ll cover redirection in more detail later.

Expected Output (data.csv):

Name,Age,Location
Alice,30,New_York
Bob,25,Los_Angeles
Charlie,35,Chicago

These examples demonstrate how simple commands can be combined to achieve powerful and useful results. Experiment with these commands and modify them to fit your specific needs!

2. Mastering Shell Techniques

The shell is the heart of your command line, acting as the bridge between you and the operating system. It does more than just run commands; it helps you manage and streamline your tasks efficiently. Let’s dive into some essential shell techniques.

Understanding the Shell with `ls`

To get a clear picture of what the shell does versus what a command does, let’s look at the ls command. When you run ls, the shell and the command work together to get things done:

Shell’s Job: The shell processes wildcards (like *.py), sets up input and output redirections, and manages environment variables.
Command’s Job: The ls command lists directory contents.

Example: List all Python files in a directory.

ls *.py

In this example:

The shell expands *.py to match all Python files (e.g., data.py, main.py).
The ls Command receives the list of Python files and displays them.

1. Pattern Matching for Filenames

Pattern matching allows you to select multiple files with a single command using wildcards.

Example 1: List all text files.

ls *.txt

This command lists all files ending with .txt, like report.txt, notes.txt, and summary.txt.

Example 2: List all files that start with “data”.

ls data*

This command lists all files starting with “data”, like data1.csv, data2.csv, and database.db.

Example 3: List all files that have numbers in their names.

ls *[0-9]*

This command lists files that contain any digit, like file1.txt, report2.doc, and data2023.csv.

2. Variables to Store Values

Variables let you store data that can be reused in your commands.

Example 1: Create a variable to store your project directory path.

PROJECT_DIR="/home/user/myproject"
cd $PROJECT_DIR

This command sets the PROJECT_DIR variable and then uses it to change to that directory.

Example 2: Store the result of a command in a variable.

DATE=$(date +%Y-%m-%d)
echo "Today's date is $DATE"

This command sets the DATE variable to the current date using command substitution and then prints the contents of the DATE variable.

3. Redirection of Input and Output

Redirection allows you to control where your command’s input and output go.

Example 1: Save the output of a command to a file.

ls -l > filelist.txt

This command lists all files in long format and saves the output to filelist.txt. Redirection with the single right bracket will replace the contents of the target file.

Example 2: Append the output of a command to a file.

echo "New entry" >> filelist.txt

This command adds “New entry” to the end of filelist.txt. Redirection with two right brackets will append to the contents of the target file.

Example 3: Use a file as input for a command.

sort < unsorted.txt

This command sorts the contents of unsorted.txt and displays the result. The single left bracket reads the contents of the file on the right (unsorted.txt in this case) and uses them as input to the command on the left (sort in this case).

4. Quoting and Escaping to Disable Shell Features

Quoting and escaping are used to handle special characters in your commands. Double quotes won’t expand wildcards or break the string on spaces, but will still expand variables and substitute commands with $(). Content in single quotes isn’t processed by the shell at all; all characters are retained literally.

Example 1: Search for a phrase with spaces in a file.

grep "my search phrase" file.txt

This command searches for the exact phrase “my search phrase” in file.txt.

Example 2: Use a variable inside a double-quoted string.

NAME="Alice"
echo "Hello, $NAME"

This command prints “Hello, Alice” using the NAME variable.

Example 3: Use single quotes to prevent variable expansion.

echo 'Hello, $NAME'

This command prints Hello, $NAME without expanding the variable.

Example 4: Use backslashes to escape special characters in a double-quoted string.

echo "This path costs \$139: C:\\Windows\\System32"

This command prints This path costs $139: C:\Windows\System32. The dollar sign and backlashes must be escaped with a \ so that they aren’t treated as special characters.

Example 5: Include a literal double quote in a string.

echo "He said, \"Hello, World!\""

This command prints He said, "Hello, World!" by escaping the double quotes.

Example 6:: Include a literal single quote in a single-quoted string.

echo 'What'\''s up?'

This command prints “What’s up?” using a more complex escape sequence—first, you close the single-quote string, then without a space you add a literal single-quote character with \', then you start a single-quoted string again. In bash, when there are no spaces between two strings, the strings will be joined together.

5. The Search Path for Locating Programs to Run

The shell uses the PATH variable to find programs to execute.

Example 1: View your current PATH.

echo $PATH

This command displays the directories in your current PATH.

Example 2: Add a directory to your PATH temporarily.

export PATH=$PATH:/home/user/bin

This command adds /home/user/bin to your PATH for the current session. It reads the PATH variable, then concatenates the string “:/home/user/bin” to the end, similarly to how we escaped a single quote.

Example 3: Add a directory to your PATH permanently.

echo 'export PATH=$PATH:/home/user/bin' >> ~/.bashrc
source ~/.bashrc

This command appends the new PATH to your .bashrc file (using redirection!). .bashrc is run every time a new shell is started. source reloads .bashrc for the current shell, using the new PATH for the current session.

6. Saving Changes to Your Shell Environment

Save environment changes by adding them to your shell configuration file.

Example 1: Set an environment variable for the current session.

export EDITOR=nano

This command sets the default text editor to nano for the current session.

Example 2: Make an environment variable permanent.

echo 'export EDITOR=nano' >> ~/.bashrc
source ~/.bashrc

This command sets the default text editor to nano by adding it to your .bashrc file.

Example 3: Create an alias for a command.

echo 'alias ll="ls -l"' >> ~/.bashrc
source ~/.bashrc

This command creates an alias ll for ls -l and makes it permanent by adding it to your .bashrc file.

Mastering these shell techniques will boost your productivity and make your command line experience more enjoyable. Dive in and start practicing these essential commands!

3. Efficient Command Recall and Editing

Tired of retyping long commands or fixing typos from scratch? The shell’s got you covered with command history and command-line editing, designed to save you time and effort.

Imagine you just ran a complex command:

md5sum *.jpg | cut -c 1-32 | sort | uniq -c | sort -nr

Need to run it again? No need to retype it. The shell keeps a history of every command you run, so you can quickly recall and rerun them. This feature, known as command history, is a favorite among power users for speeding up their workflow and cutting down on repetitive typing.

Accidentally typed *.jg instead of *.jpg?

md5sum *.jg | cut -c 1-32 | sort | uniq -c | sort -nr

No problem! Instead of starting over, just edit the command directly. The shell’s command-line editing lets you fix typos on the fly, similar to using a text editor.

In this section, you’ll discover how to leverage command history and command-line editing to enhance your efficiency and minimize errors.

Command History

The shell keeps a record of every command you execute, allowing you to recall and reuse them with ease.

Viewing the Command History

The history command lists all previously executed commands in your shell session. Each command is assigned an ID number for easy reference.

Example:

history

Output:

1000  cd $HOME/Music
1001  ls
1002  mv jazz.mp3 jazzy-song.mp3
1003  play jazzy-song.mp3
...
1481  cd
1482  firefox https://google.com
1483  history

To view the most recent commands, limit the output with a number.

Example:

history 3

Output:

1482  firefox https://google.com
1483  history
1484  history 3

You can also filter the history to find specific commands by piping to grep.

Example:

history | grep "cd"

Output:

1000  cd $HOME/Music
1092  cd ..
1123  cd Finances
1375  cd Checking
1481  cd

Environment Variables for History

Environment variables control how your shell handles command history. Key variables include HISTSIZE, HISTFILESIZE, and HISTCONTROL.

HISTSIZE: Determines the number of commands to remember in the current session.

Example:

echo $HISTSIZE

Output:

To change the size:

export HISTSIZE=1000

HISTFILESIZE: Sets the number of commands to save in the history file.

Example:

echo $HISTFILESIZE

Output:

To change the file size:

export HISTFILESIZE=2000

HISTCONTROL: Controls what commands are saved. Common values include ignorespace (ignore commands starting with a space) and ignoredups (ignore duplicate commands).

Example:

export HISTCONTROL=ignoredups

To combine options:

export HISTCONTROL=ignoredups:ignorespace

HISTIGNORE: Excludes specific commands from being saved.

Example:

export HISTIGNORE="ls:cd:pwd"

This setting prevents ls, cd, and pwd commands from being saved in the history.

Note that you should add these export commands to your .bashrc for them to take effect outside the current shell.

History Expansion

History expansion is a powerful shell feature that lets you reuse and modify previous commands quickly. It uses special expressions, typically starting with an exclamation mark (!), to reference commands from your history.

Basic History Expansion

You can repeat the last command with !!.

Example:

echo "Hello, World!"
!!

This reruns echo "Hello, World!".

Referencing Specific Commands

To repeat a specific command from your history, use its ID number.

Example:

!1002

This reruns the command with ID 1002 (mv jazz.mp3 jazzy-song.mp3 in our example).

Using Substring Matches

You can recall the most recent command that starts with a specific string.

Example:

!mv

This reruns the most recent mv command.

To recall a command containing a specific substring anywhere, use !?string?.

Example:

!?cd?

This reruns the most recent command containing cd.

Modifying Previous Commands

You can also modify the previous command before running it. For instance, use ^old^new to replace the first occurrence of old with new in the last command.

Example:

echo "Hello, Alice"
^Alice^Bob

This changes Alice to Bob, rerunning the command as echo "Hello, Bob".

Using `!$` and `!*`

!$ refers to the last argument of the previous command.

Example:

ls /some/directory
cd !$

This changes to /some/directory.

!* refers to all arguments of the previous command.

Example:

cp file1.txt file2.txt /backup/
rm !*

This removes both file1.txt and file2.txt from /backup/.

Safety with `:p`

To avoid mistakes, use :p to print the command without executing it.

Example:

!1002:p

This prints the command with ID 1002 without running it. You can then confirm it before rerunning it.

Command-Line Editing

Command-line editing is essential for fixing mistakes, modifying commands, and creating new commands efficiently. Here are some key techniques to help you edit commands quickly and effectively.

Cursoring Within a Command

Use the left and right arrow keys to move the cursor back and forth within a command line. This allows you to make changes without retyping the entire command.

Example: To correct a typo, press the left arrow key to move the cursor to the mistake, fix it, and then press Enter.

Useful Keystrokes:

Left arrow: Move left by one character
Right arrow: Move right by one character
Ctrl + left arrow: Move left by one word (on macOS, try Option + left arrow)
Ctrl + right arrow: Move right by one word (on macOS, try Option + right arrow)
Ctrl + a: Move to the beginning of the command line
Ctrl + e: Move to the end of the command line

Incremental Search

Press Ctrl + r to initiate an incremental search through your command history. This lets you search for a previously run command without scrolling through the entire history.

Example: Press Ctrl + r and start typing part of a previous command. The shell will display matching commands as you type. Press Ctrl + r again to cycle through other matches. Once you find the command, press Enter to run it or use the arrow keys to edit it.

Additional Tips:

To recall the most recent search string, press Ctrl + r twice.
To stop the search and continue editing the current command, press Esc, Ctrl + j, or any arrow key.
To quit the search and clear the command line, press Ctrl + g or Ctrl + c.

By mastering these command-line editing techniques, you can save time, reduce errors, and improve your efficiency when working in the shell.

4. Navigating the Filesystem with Speed

Navigating the filesystem efficiently can significantly boost productivity. Here are some techniques:

Using `cd`

The cd (change directory) command is fundamental for navigation. Here are some practical tips:

Jump to the Home Directory:
```
cd
```
This command takes you directly to your home directory, regardless of your current location in the filesystem.
Navigate Using Variables:
```
cd $HOME/Work
cd ~/Work
```
Both $HOME and ~ are shortcuts to your home directory, making it easy to navigate to subdirectories within it.
Navigate to Another User’s Home Directory:
```
cd ~username
```

CDPATH

Set the CDPATH variable to define base directories for quick navigation.

Setting CDPATH:
```
export CDPATH=$HOME:$HOME/Work
```
Using CDPATH:
```
cd Work
```

Organizing Your Home Directory

A well-organized home directory structure can streamline your workflow.

Use Descriptive Names:

mkdir -p $HOME/Projects/{Work,Personal,OpenSource}

Create Aliases:

alias work="cd $HOME/Projects/Work"
alias personal="cd $HOME/Projects/Personal"

Using pushd and popd for Directory Management

The pushd and popd commands are used to manage a stack of directories, allowing for efficient navigation between them. Here’s how they work:

pushd: Adds a directory to the stack and changes to that directory.
popd: Removes the top directory from the stack and changes to the new top directory.

Basic Usage

Pushing a Directory:
```
pushd /path/to/directory
```
This command adds /path/to/directory to the stack and changes the current directory to it.
Popping a Directory:
```
popd
```
This command removes the top directory from the stack and changes the current directory to the new top directory.

Example Workflow

Let’s say you start in your home directory (~/) and execute the following commands:

Pushing Directories:

pushd /var/www/html
pushd /etc/apache2

The stack now looks like this:

~/           (top)
/etc/apache2
/var/www/html

Popping Directories:
```
popd
```
After this command, the stack looks like this:
```
~/           (top)
/var/www/html
```
Navigating with Aliases:

You can create aliases for pushd and popd to make navigation easier:
```
alias pd="popd"
alias gd="pushd"
```
Now you can use gd to push directories onto the stack and pd to pop directories off the stack.

These techniques will help you navigate your filesystem efficiently, allowing you to focus more on your tasks and less on typing long paths.

Conclusion

Mastering the command line can transform your productivity and efficiency. We’ve covered combining commands with pipes, mastering shell techniques, recalling and editing commands efficiently, and navigating the filesystem with speed. Dive in, experiment, and watch your skills grow!

For further reading, I highly recommend the book “Efficient Linux at the Command Line”. It took my command-line skills to the next level and inspired many of the tips and tricks shared in this blog.

Consolidating Multiple SFTP Accounts Into One Master Account

2020-03-16T00:00:00+00:00

Photo by Dan Meyers

Recently, a client implemented a data-intensive workflow to generate various reports and insights from a list of facilities as part of an EpiTrax installation. Because a significant portion of these files contain sensitive healthcare data, they needed to strictly comply with HIPAA. Optimally, facilities should be able to transfer files securely and exclusively to our server. One of the best methods of achieving this is to create individual SSH File Transfer Protocol (SFTP) accounts for each source.

SFTP account

Private SFTP accounts were established for each facility and the data was received at a designated path. At these individual points of contact, a third-party application picks up the data and processes further into the pipeline. The following demonstrates how SFTP accounts are developed and configured:

Create a user group for SFTP accounts:

$ addgroup sftpusers

Configure the following settings in sshd_config (this enables an SFTP account and sets the default location as the home path):

$ vi /etc/ssh/sshd_config
...
# override default of no subsystems
Subsystem       sftp    internal-sftp...

Match Group sftpusers
    ChrootDirectory /home/%u
    AllowTCPForwarding no
    X11Forwarding no
    ForceCommand internal-sftp

Restart SSH server to apply changes:

$ systemctl restart ssh

Create an SFTP user account for a facility and place in a folder on the home path to receive data:

# set new user name
sftpuser=the-new-username
useradd $sftpuser
usermod -g sftpusers -s /usr/sbin/nologin $sftpuser
mkdir -p /home/$sftpuser/INPUT_PATH/
chown -R root:root /home/$sftpuser

Mount multiple accounts to one account

The goal here is to point the data from many facilities to one location, but using a single account and path for multiple sites’ data could result in a breach of security and/or privacy. Mounting the receiving path of a facility’s data onto a single master account and then to a “mount point” with a unique facility name takes care of this issue. The process next consolidates files from individual paths on a master account in one place where the application picks up messages for further processing.

The SFTP accounts and the master account should be attached to the same group. This will permit individual accounts to write on the master account-mounted path. In turn, the master account can read files from the same location. This location now has administrative rights for both the SFTP user the group. Group permission of the mounted folder is set to sftpgroup and user permission is set to the facility account.

# create master user account
$ adduser master
$ passwd master

# Add master user to sftpgroup group
usermod -a -G sftpgroup master
getent group sftpgroup

# MOUNT_PATH and sub folders
mkdir -p /home/master/MOUNT_PATH/{Input,Pickup,Backup,Archive}
chown -R master:master /home/master/MOUNT_PATH

We wrote a script to automate the process of creating an SFTP account, mounting it at the master account path, and adding fstab entries to save the mount in the case of a reboot. The script not only saves time, but also avoids human error when creating accounts for all facilities.

#!/bin/bash

# ./create_sftp_account.sh sftpfacilityone FACILITY_ONE

sftpuser=$1
facility_name=$2

# Create SFTP account and add to sftpgroup
useradd $sftpuser
usermod -g sftpusers -s /usr/sbin/nologin $sftpuser
usermod -a -G sftpgroup $sftpuser
mkdir -p /home/$sftpuser/INPUT_PATH
chown root:root /home/$sftpuser

# Create path specific to facility in master account Input folder
mkdir -p /home/master/MOUNT_PATH/Input/$facility_name
chown -R $sftpuser:sftpgroup /home/master/MOUNT_PATH/Input/$facility_name

# Mount sftp account into master account and set permissions
mount --bind /home/master/MOUNT_PATH/Input/$facility_name /home/$sftpuser/INPUT_PATH
chown -R $sftpuser:sftpgroup /home/$sftpuser/INPUT_PATH
chmod g=rwxs /home/$sftpuser/INPUT_PATH

# Add fstab entry to persist and mount on reboot
echo "/home/master/MOUNT_PATH/Input/$facility_name /home/$sftpuser/INPUT_PATH none bind 0 0" >> /etc/fstab
echo "Created user $sftpuser at $facility_name mount point successfully"

Files at one location

Now, data files from facilities are available at individual folders under MOUNT_PATH/Input on the master account. This enables third-party applications to pick up files in a straightforward way to proceed with further processing. It also helps our client access the files for review from the master account easily without navigating into each separate account.

Summary

Mounting multiple SFTP accounts onto one master account turns out to be an efficient and beneficial method of consolidating data. Both safe and secure, running separate SFTP accounts establishes an exclusive private link between facilities and servers. The master account has the unique ability to access files belonging to each facility in order to process the data further.

Tip: In order to avoid broken mounts, check the status by using the command mount -fav. Problems with mount configurations can cause broken mounts after rebooting the server.

Making sense of XML/JSON items in the shell

2019-12-31T00:00:00+00:00

Working as a system administrator means I have to spend quite some time during my work (and even during casual surfing) with the terminal. Sometimes I feel that certain information I want could just be fetched and parsed through the terminal, without having to use my mouse and point to the browser.

Some of the websites I visit use XML and JSON, which we could parse with Bash scripting. Previously I wrote a Ruby script to call Nokogiri to parse the XML elements until I found a Bash tool that could do the same thing.

These tools have already been around for quite a while—I’d just like to share what I did with them. The tools I used are xmlstarlet for XML parsing and jq for JSON.

XML

I have the following XML elements, and I’ll save them to a file called data.xml:

<rss version="2.0"
    xmlns:dc="http://purl.org/dc/elements/1.1/"
    xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
    xmlns:admin="http://webns.net/mvcb/"
    xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
    xmlns:content="http://purl.org/rss/1.0/modules/content/">

    <channel>
        <title>eSolat JAKIM : Waktu Solat Hari Ini</title>
        <link>Gombak,Petaling,Sepang,Hulu Langat,Hulu Selangor,Rawang,S.Alam</link>
        <description>Gombak,Petaling,Sepang,Hulu Langat,Hulu Selangor,Rawang,S.Alam</description>
        <dc:language>ms</dc:language>
        <dc:creator>www.e-solat.gov.my</dc:creator>
        <dc:rights>Copyright JAKIM</dc:rights>
        <dc:date>26-12-2019 00:37:31</dc:date>
        <admin:generatorAgent rdf:resource="expressionengine" />

        <item>
            <title>Imsak</title>
            <description>05:53:00</description>
        </item>
        <item>
            <title>Subuh</title>
            <description>06:03:00</description>
        </item>
        <item>
            <title>Syuruk</title>
            <description>07:14:00</description>
        </item>
        <item>
            <title>Zohor</title>
            <description>13:16:00</description>
        </item>
        <item>
            <title>Asar</title>
            <description>16:39:00</description>
        </item>
        <item>
            <title>Maghrib</title>
            <description>19:13:00</description>
        </item>
        <item>
            <title>Isyak</title>
            <description>20:28:00</description>
        </item>
    </channel>
</rss>

I’ll use xmlstarlet, together with a bunch of the related flags to parse these elements into something which is more eye-friendly.

xmlstarlet sel -T -t -n \
  -o "------------------------------" -n \
  -o "Area: " \
  -m "//channel" -v "link" -n  \
  -o "Date: " \
  -m "//channel" -v "dc:date" -n  \
  -o "------------------------------" -n \
  -t -m "//item" -o "Time: " -v "title" \
  -o " -- " \
  -o "Time: " -v "description" -n  \
  /path/to/data.xml

The output looks like this:

------------------------------
Area: Gombak,Petaling,Sepang,Hulu Langat,Hulu Selangor,Rawang,S.Alam
Date: 26-12-2019 00:37:31
------------------------------
Time: Imsak -- Time: 05:53:00
Time: Subuh -- Time: 06:03:00
Time: Syuruk -- Time: 07:14:00
Time: Zohor -- Time: 13:16:00
Time: Asar -- Time: 16:39:00
Time: Maghrib -- Time: 19:13:00
Time: Isyak -- Time: 20:28:00

I’ll put this in a Bash script, and call it xmlstarlet-time.sh.

#!/bin/bash

XMLPATH=/home/seth/data.xml

if [[ -z $1 ]]; then
  echo "Put the location code"
  echo "$0 <location code>"
  echo -n
  exit
fi

lynx -source "https://www.e-solat.gov.my/index.php?r=esolatApi/xmlfeed&zon=$1" > $XMLPATH

xmlstarlet sel -T -t -n \
  -o "------------------------------" -n \
  -o "Area: " -m "//channel" -v "link" -n  \
  -o "Date: " -m "//channel" -v "dc:date" -n  \
  -o "------------------------------" -n \
  -t -m "//item" -o "Time: " -v "title" -o " -- " -o "Time: " -v "description" -n  \
  $XMLPATH

Now, after making it executable with chmod +x xmlstarlet-time.sh, I can just run the script whenever I need the info. In my case, I would type ./xmlstarlet-time.sh SGR01 in order to get the above information. I got the code (in my case) from the XML URL above. Your use case will likely be different.

JSON

Let’s say I want to grab the latest currency exchange, taking the base of USD from exchangeratesapi.io. I can use curl to get the data.

$ curl -s 'https://api.exchangeratesapi.io/api/latest?base=USD'

Which will return:

{"rates":{"CAD":1.3160649819,"HKD":7.7879061372,"ISK":122.3826714801,"PHP":50.8402527076,"DKK":6.7429602888,"HUF":299.4223826715,"CZK":23.0009025271,"GBP":0.7719584838,"RON":4.3131768953,"SEK":9.4361913357,"IDR":13985.0180505415,"INR":71.2567689531,"BRL":4.0835740072,"RUB":62.0877256318,"HRK":6.719765343,"JPY":109.3772563177,"THB":30.155234657,"CHF":0.9817689531,"EUR":0.9025270758,"MYR":4.1364620939,"BGN":1.7651624549,"TRY":5.9561371841,"CNY":7.0074909747,"NOK":8.94566787,"NZD":1.5086642599,"ZAR":14.1935018051,"USD":1.0,"MXN":18.9626353791,"SGD":1.3553249097,"AUD":1.4457581227,"ILS":3.4714801444,"KRW":1162.1931407942,"PLN":3.8445848375},"base":"USD","date":"2019-12-24"}

Using jq, we can format the information more readably:

$ curl -s 'https://api.exchangeratesapi.io/api/latest?base=USD' | jq
{
  "rates": {
    "CAD": 1.3160649819,
    "HKD": 7.7879061372,
    "ISK": 122.3826714801,
    "PHP": 50.8402527076,
    "DKK": 6.7429602888,
    "HUF": 299.4223826715,
    "CZK": 23.0009025271,
    "GBP": 0.7719584838,
    "RON": 4.3131768953,
    "SEK": 9.4361913357,
    "IDR": 13985.0180505415,
    "INR": 71.2567689531,
    "BRL": 4.0835740072,
    "RUB": 62.0877256318,
    "HRK": 6.719765343,
    "JPY": 109.3772563177,
    "THB": 30.155234657,
    "CHF": 0.9817689531,
    "EUR": 0.9025270758,
    "MYR": 4.1364620939,
    "BGN": 1.7651624549,
    "TRY": 5.9561371841,
    "CNY": 7.0074909747,
    "NOK": 8.94566787,
    "NZD": 1.5086642599,
    "ZAR": 14.1935018051,
    "USD": 1,
    "MXN": 18.9626353791,
    "SGD": 1.3553249097,
    "AUD": 1.4457581227,
    "ILS": 3.4714801444,
    "KRW": 1162.1931407942,
    "PLN": 3.8445848375
  },
  "base": "USD",
  "date": "2019-12-24"
}

Next, I can make use of the tool in my shell script.

#!/bin/bash -l

RED='\033[0;31m'
YELLOW='\033[1;33m'
GREEN='\033[0;32m'
NC='\033[0m'

if [ -z `which jq` ]; then
  printf "You need to install jq, a JSON parsing tool \n"
  exit
fi

sourcemoney=$(echo $2 | tr '[:lower:]' '[:upper:]')
target=$(echo $3 | tr '[:lower:]' '[:upper:]')

sumber=$(curl -s "https://api.exchangeratesapi.io/api/latest?base=$sourcemoney" | jq . | grep -i $target | awk -F ':|,' '{ print $2 }')

jumlah=$(printf "%f*%f\n" $1 $sumber | bc)

printf "Price per unit: ${GREEN}1 $sourcemoney${NC} = ${YELLOW}$target %.2f${NC}\n" $sumber

echo -e "Source money: ${YELLOW}$sourcemoney $1${NC}"
echo -n

printf "Total money after the conversion: ${YELLOW}$target %.2f ${NC}\n" $jumlah

Then I can save the script into a file called moneychanger-with-api.sh and make it executable with chmod +x moneychanger-with-api.sh.

And now the script will do the parsing for me, without the need for a browser.

$ ./moneychanger-with-api.sh 100 usd eur
Price per unit: 1 USD =  EUR 0.90
Source money: USD 100
Total money after the conversion: EUR 90.25

$ ./moneychanger-with-api.sh 100 eur sgd
Price per unit: 1 EUR =  SGD 1.50
Source money: EUR 100
Total money after the conversion: SGD 150.17

Shell Command Outputs Truncated in Python

2018-04-05T00:00:00+00:00

Photo by Sarah Pflug of Burst

Recently I was working on a Python script to do some parsing and processing on the output of shell commands in Ubuntu. The output that showed up was truncated.

The below sections will walk through the debugging process to identify the root cause and implement a solution with detailed explanation, using Python 2.

Problem

The following code block shows the output of a shell command which lists the installed packages, name and version, in Ubuntu.

# dpkg -l | grep ^ii | awk '{print $2 "    " $3}'
accountsservice    0.6.35-0ubuntu7.3
acl    2.2.52-1
adduser    3.113+nmu3ubuntu3
ant    1.9.3-2build1
ant-optional    1.9.3-2build1
apache2    2.4.7-1ubuntu4.18
apache2-bin    2.4.7-1ubuntu4.18
apache2-data    2.4.7-1ubuntu4.18
apache2-utils    2.4.7-1ubuntu4.18
apparmor    2.10.95-0ubuntu2.6~14.04.1

The same shell command executes in the Python console but the output shows truncated values for a few packages’ versions, for example, accountsservice, adduser, apache2, etc.

>>> import subprocess
>>> installed_packages = subprocess.check_output(['dpkg -l | grep ^ii | awk \'{print $2 "    " $3}\''], shell=True)
>>> print installed_packages
accountsservice    0.6.35-0ubuntu7.
acl    2.2.52-1
adduser    3.113+nmu3ubuntu
ant    1.9.3-2build1
ant-optional    1.9.3-2build1
apache2    2.4.7-1ubuntu4.1
apache2-bin    2.4.7-1ubuntu4.1
apache2-data    2.4.7-1ubuntu4.1
apache2-utils    2.4.7-1ubuntu4.1
apparmor    2.10.95-0ubuntu2

Root Cause

To identify the root cause of the problem, I started with source command dpkg -l command without any filters and processing. I have noticed two different results for this command, with and without less command. The less command showed the complete result with scrolling as below.

# dpkg -l | less
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                                  Version                                    Architecture Description
+++-=====================================-==========================================-============-===============================================================================
rc  aacraid                               1.2.1-52011                                amd64        This driver supports Adaptec by PMC aacraid family of cards.
ii  accountsservice                       0.6.35-0ubuntu7.3                          amd64        query and manipulate user account information
ii  acl                                   2.2.52-1                                   amd64        Access control list utilities
ii  adduser                               3.113+nmu3ubuntu3                          all          add and remove users and groups
ii  ant                                   1.9.3-2build1                              all          Java based build tool like make
ii  ant-optional                          1.9.3-2build1                              all          Java based build tool like make - optional libraries
ii  apache2                               2.4.7-1ubuntu4.18                          amd64        Apache HTTP Server
ii  apache2-bin                           2.4.7-1ubuntu4.18                          amd64        Apache HTTP Server (binary files and modules)
ii  apache2-data                          2.4.7-1ubuntu4.18                          all          Apache HTTP Server (common files)
ii  apache2-utils                         2.4.7-1ubuntu4.18                          amd64        Apache HTTP Server (utility programs for web servers)
rc  apache2.2-common                      2.2.22-1ubuntu1.11                         amd64        Apache HTTP Server common files
ii  apparmor                              2.10.95-0ubuntu2.6~14.04.1                 amd64        user-space parser utility for AppArmor

But dpkg -l prints on the screen with truncated data due to the columns width constraint. The truncated values exactly match the Python console output. The output column width is decided by environment variable COLUMNS and commands restrict the column width in output based on COLUMNS value.

# echo $COLUMNS
127

# dpkg -l
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                   Version          Architecture     Description
+++-======================-================-================-==================================================
rc  aacraid                1.2.1-52011      amd64            This driver supports Adaptec by PMC aacraid family
ii  accountsservice        0.6.35-0ubuntu7. amd64            query and manipulate user account information
ii  acl                    2.2.52-1         amd64            Access control list utilities
ii  adduser                3.113+nmu3ubuntu all              add and remove users and groups
ii  ant                    1.9.3-2build1    all              Java based build tool like make
ii  ant-optional           1.9.3-2build1    all              Java based build tool like make - optional librari
ii  apache2                2.4.7-1ubuntu4.1 amd64            Apache HTTP Server
ii  apache2-bin            2.4.7-1ubuntu4.1 amd64            Apache HTTP Server (binary files and modules)
ii  apache2-data           2.4.7-1ubuntu4.1 all              Apache HTTP Server (common files)
ii  apache2-utils          2.4.7-1ubuntu4.1 amd64            Apache HTTP Server (utility programs for web serve
rc  apache2.2-common       2.2.22-1ubuntu1. amd64            Apache HTTP Server common files
ii  apparmor               2.10.95-0ubuntu2 amd64            user-space parser utility for AppArmor

Solution

The subprocess module of Python provides complete untruncated output of the shell command when the argument env={} is passed to check_output function:

>>> installed_packages = subprocess.check_output(['dpkg -l | grep ^ii | awk \'{print $2 "    " $3}\''], shell=True, env={})
>>> print installed_packages
accountsservice    0.6.35-0ubuntu7.3
acl    2.2.52-1
adduser    3.113+nmu3ubuntu3
ant    1.9.3-2build1
ant-optional    1.9.3-2build1
apache2    2.4.7-1ubuntu4.18
apache2-bin    2.4.7-1ubuntu4.18
apache2-data    2.4.7-1ubuntu4.18
apache2-utils    2.4.7-1ubuntu4.18
apparmor    2.10.95-0ubuntu2.6~14.04.1

Explanation

Curious to know what is happening behind the scenes? The check_output function uses C library functions execv or execve for processing. It chooses the function based on the env argument.

Reference:

When no env argument is passed to subprocess.check_output, the os.execv function is called.

When an env argument is passed to subprocess.check_output, the os.execve function is called.

for (i = 0; exec_array[i] != NULL; ++i) {
    const char *executable = exec_array[i];
    if (envp) {
        execve(executable, argv, envp);
    } else {
        execv(executable, argv);
    }

What makes the execv and execve functions produce different output?

The execv function passes through the shell COLUMNS variable which leads to truncating output columns to 127 width, like our reference system.

# echo $COLUMNS
127

>>> print subprocess.check_output(['dpkg -l | grep libqtcore4'], shell=True)
ii  libqtcore4:amd64          4:4.8.5+git192-g0 amd64             Qt 4 core module

>>> print subprocess.check_output(['dpkg -l | grep libqtcore4'], shell=True, env={'COLUMNS':'127'})
ii  libqtcore4:amd64          4:4.8.5+git192-g0 amd64             Qt 4 core module

The execve function uses additional argument environment variables and it is based on the environ function. It uses environment variables available in env command which doesn’t have COLUMNS initialised. So output values returned without any column width restriction.

>>> print subprocess.check_output(['dpkg -l | grep libqtcore4'], shell=True, env={})
ii  libqtcore4:amd64                      4:4.8.5+git192-g085f851+dfsg-2ubuntu4.1    amd64        Qt 4 core module

>>> print subprocess.check_output(['dpkg -l | grep libqtcore4'], shell=True, env={'COLUMNS':''})
ii  libqtcore4:amd64                      4:4.8.5+git192-g085f851+dfsg-2ubuntu4.1    amd64        Qt 4 core module

For more details refer to the man pages of execv, execve, environ.

Conclusion

It is always good to pass env={} argument to subprocess.check_output function whenever processing shell command output in Python. It helps avoid unstable results down the line due to truncated values.

How to check process duration in Linux with the “ps” command

2017-08-08T00:00:00+00:00

In certain cases we might want to get a certain process’ elapsed time for our own reason. Turns out “ps” command could easily assist us in that. According to “ps” manual, etime could put the duration of time in [[DD-]hh:]mm:ss. format, while etimes in seconds.

From “ps” manpage:

etime       ELAPSED   elapsed time since the process was started, in the form [[DD-]hh:]mm:ss.
etimes      ELAPSED   elapsed time since the process was started, in seconds.

To use that, we could use (in [[DD-]hh:]mm:ss. format):

ps -p "pid" -o etime

or in seconds:

ps -p "pid" -o etimes

In this case the “pid” should be replaced with your intended process ID.

The following will help to nicely reporting the output. We can put -o etime or -o etimes with other argument, that is “command”, in order to show the executed command along with its very own absolute path:

ps -p "28590" -o etime,command

ELAPSED COMMAND
21:45 /usr/bin/perl ./fastcgi-wrapper.pl 7999

We can also get the start date of the process’ execution:

najmi@ubuntu-ampang:~$ ps -p 21745 -o etime,command,start
    ELAPSED COMMAND                      STARTED
 1-19:47:45 /usr/lib/firefox/firefox      Aug 02

What if we do not want to manually parsing the PID, instead (since we are very sure) to just get the name of the running application? We could just simply use pgrep or pidof

najmi@ubuntu-ampang:~$ ps -p $(pgrep firefox) -o pid,cmd,start,etime
  PID CMD                          STARTED     ELAPSED
21745 /usr/lib/firefox/firefox      Aug 02  2-04:29:36

najmi@ubuntu-ampang:~$ ps -p $(pidof firefox) -o pid,cmd,start,etime
  PID CMD                          STARTED     ELAPSED
21745 /usr/lib/firefox/firefox      Aug 02  2-04:29:42

What if the command issued many processes? Take an example of the Chrome browser:

najmi@ubuntu-ampang:~$ ps -p $(pidof chrome) -o pid,comm,cmd,start,etime
error: process ID list syntax error

Usage:
 ps [options]

 Try 'ps --help <simple|list|output|threads|misc|all>'
  or 'ps --help <s|l|o|t|m|a>'
 for additional help text.

For more details see ps(1).
</s|l|o|t|m|a></simple|list|output|threads|misc|all>

The best way (so far) that I could get is by creating a loop. It seems pidof is much more accurate when parsing the exact application (string) that we feed into it.

With pgrep:

najmi@ubuntu-ampang:~$ for i in `pgrep chrome`;do ps -p $i -o pid,comm,cmd,start,etime|tail -n +2;done
 2255 chrome          /opt/google/chrome/chrome - 08:05:43    02:55:17
 4990 chrome          /opt/google/chrome/chrome - 10:39:16       21:44
 5567 chrome          /opt/google/chrome/chrome - 10:53:13       07:47
 9448 chrome          /opt/google/chrome/chrome -   Jul 31  3-12:25:08
10033 chrome          /opt/google/chrome/chrome     Jul 27  8-10:43:42
10044 chrome          /opt/google/chrome/chrome -   Jul 27  8-10:43:42
10050 chrome          /opt/google/chrome/chrome -   Jul 27  8-10:43:42
10187 chrome          /opt/google/chrome/chrome -   Jul 27  8-10:43:39
10234 chrome          /opt/google/chrome/chrome -   Jul 27  8-10:43:37
10236 chrome          /opt/google/chrome/chrome -   Jul 27  8-10:43:37
19440 chrome          /opt/google/chrome/chrome - 22:30:34    12:30:26
20229 chrome          /opt/google/chrome/chrome -   Aug 03  1-09:51:25
20514 chrome          /opt/google/chrome/chrome - 22:52:25    12:08:35
20547 chrome          /opt/google/chrome/chrome - 22:52:36    12:08:24
21009 chrome          /opt/google/chrome/chrome -   Aug 03  1-09:27:11
22458 chrome          /opt/google/chrome/chrome -   Jul 27  8-03:44:07
22474 chrome-gnome-sh /usr/bin/python3 /usr/bin/c   Jul 27  8-03:44:07
23681 chrome          /opt/google/chrome/chrome -   Aug 03  1-03:33:45
23691 chrome          /opt/google/chrome/chrome -   Aug 03  1-03:33:45
23870 chrome          /opt/google/chrome/chrome - 00:15:15    10:45:45
24544 chrome          /opt/google/chrome/chrome - 00:40:17    10:20:43
25116 chrome          /opt/google/chrome/chrome - 00:51:31    10:09:29
25466 chrome          /opt/google/chrome/chrome - 00:59:55    10:01:05
29060 chrome          /opt/google/chrome/chrome - 02:15:42    08:45:18

With pidof:

najmi@ubuntu-ampang:~$ for i in `pidof chrome`;do ps -p $i -o pid,comm,cmd,start,etime|tail -n +2;done
29060 chrome          /opt/google/chrome/chrome - 02:15:42    08:47:40
25466 chrome          /opt/google/chrome/chrome - 00:59:55    10:03:27
25116 chrome          /opt/google/chrome/chrome - 00:51:31    10:11:51
24544 chrome          /opt/google/chrome/chrome - 00:40:17    10:23:05
23870 chrome          /opt/google/chrome/chrome - 00:15:15    10:48:07
23691 chrome          /opt/google/chrome/chrome -   Aug 03  1-03:36:07
23681 chrome          /opt/google/chrome/chrome -   Aug 03  1-03:36:07
22458 chrome          /opt/google/chrome/chrome -   Jul 27  8-03:46:29
21009 chrome          /opt/google/chrome/chrome -   Aug 03  1-09:29:33
20547 chrome          /opt/google/chrome/chrome - 22:52:36    12:10:46
20514 chrome          /opt/google/chrome/chrome - 22:52:25    12:10:57
20229 chrome          /opt/google/chrome/chrome -   Aug 03  1-09:53:47
19440 chrome          /opt/google/chrome/chrome - 22:30:34    12:32:48
10236 chrome          /opt/google/chrome/chrome -   Jul 27  8-10:45:59
10234 chrome          /opt/google/chrome/chrome -   Jul 27  8-10:45:59
10187 chrome          /opt/google/chrome/chrome -   Jul 27  8-10:46:01
10050 chrome          /opt/google/chrome/chrome -   Jul 27  8-10:46:04
10044 chrome          /opt/google/chrome/chrome -   Jul 27  8-10:46:04
10033 chrome          /opt/google/chrome/chrome     Jul 27  8-10:46:04
 9448 chrome          /opt/google/chrome/chrome -   Jul 31  3-12:27:30
 5567 chrome          /opt/google/chrome/chrome - 10:53:13       10:09
 4990 chrome          /opt/google/chrome/chrome - 10:39:16       24:06
 2255 chrome          /opt/google/chrome/chrome - 08:05:43    02:57:39

There is other tool, called as stat which records the timestamp of a file but for slightly different purpose. Stay tuned for the next blogpost!

Age comparison in Bash for files and processes

2017-05-22T00:00:00+00:00

You want your script to run a command only if elapsed-time for a given process is greater than X?

Well, bash does not inherently understand a time comparison like:

if [ 01:23:45 -gt 00:05:00 ]; then
    foo
fi

However, bash can compare timestamps of files using -ot and -nt for “older than” and “newer than”, respectively. If the launch of our process includes creation of a PID file, then we are in luck! At the beginning of our loop, we can create a file with a specific age and use that for quick and simple comparison.

For example, if we only want to take action when the process we care about was launched longer than 24 hours ago, try:

touch -t $(date --date=yesterday +%Y%m%d%H%M.%S) $STAMPFILE

Then, within your script loop, compare the PID file with the $STAMPFILE, like this:

if [ $PIDFILE -ot $STAMPFILE ]; then
    foo
fi

And of course if you want to be sure you’re working with the PID file of a process which is actually responding, you can try to send it signal 0 to check:

if kill -0 `cat $PIDFILE`; then
    foo
fi

Shell efficiency: mkdir and mv

2017-03-10T00:00:00+00:00

Little tools can be a nice improvement. Not everything needs to be thought-leaderish.

For example, once upon a time in my Unix infancy I didn’t know that mkdir has the -p option to make intervening directories automatically. So back then, in order to create the path a/b/c/ I would’ve run:

mkdir a; mkdir a/b; mkdir a/b/c

when I could instead have simply run:

mkdir -p a/b/c

In working at the shell, particularly on my own local machine, I often find myself wanting to move one or several files into a different location, to file them away. For example:

mv -i ~/Downloads/Some\ Long\ File\ Name.pdf ~/some-other-long-file-name.tar.xz ~/archive/new...

at which point I realize that the subdirectory of ~/archive that I want to move those files into does not yet exist.

I can’t simply move to the beginning of the line and change mv to mkdir -p without removing my partially-typed ~/archive/new....

I can go ahead and remove that, and then after I run the command I have to change the mkdir back to mv and add back the ~/archive/new....

In one single day I found I was doing that so often that it became tedious, so I re-read the GNU coreutils manpage for mv to see if there was a relevant option I had missed or a new one that would help. And I searched the web to see if a prebuilt tool is out there, or if anyone had any nice solutions.

To my surprise I found nothing suitable, but I did find some discussion forums full of various suggestions and many brushoffs and ill-conceived suggestions that either didn’t work for me or seemed much overengineered.

The solution I came up with was very simple. I’ve been using it for a few months and am happy enough with it to share it and see if it helps anyone else.

In zsh (my main local shell) add to ~/.zshrc:

mkmv() {
    mkdir -p -- "$argv[-1]"
    mv "$@"
}

And in bash (which I use on most of the many servers I access remotely) add to ~/.bashrc:

mkmv() {
    mkdir -p -- "${!#}"
    mv "$@"
}

To use: Once you realize you’re about to try to move files or directories into a nonexistent directory, simply go to the beginning of the line (^A = control-A in standard emacs keybindings) and type mk in front of the mv that was already there:

mkmv -i ~/Downloads/Some\ Long\ File\ Name.pdf ~/some-other-long-file-name.tar.xz ~/archive/new...

It creates the directory (or directories) and then completes the move.

There are a few important considerations that I didn’t foresee in my initial naive implementation:

Having the name be somethingmv meant less typing than something requiring me to remove the mv.
For me, it needs to support not just moving one thing to one destination, but rather a whole list of things. That meant accessing the last argument (the destination) for the mkdir.
I also needed to allow through arguments to mv such as -i, -v, and -n, which I often use.
The -- argument to mkdir ensures that we don’t accidentally end up with any other options and that we can handle a destination with a leading - (which does occasionally come up).
The mv command needs to have a double-quoted "$@" so that the original parameters are each expanded into double-quoted arguments, allowing for spaces and other shell metacharacters in the paths. (See the zsh and bash manpages for details on the important difference in behavior of "$@" compared to "$*" and either of them unquoted.)

This doesn’t support GNU extensions to mv such as the option --target-directory that precedes the source paths. I don’t use that interactively, so I don’t mind.

Because this is such a small thing, I avoided for years bothering to set it up. But now that I use it all the time, I’m glad I have it!

Using Awk to beautify grep searches

2017-01-18T00:00:00+00:00

Recently we’ve seen a sprout of re-implementations of many popular Unix tools. With the expansion of communities built around new languages or platforms, it seems that apart from the novelties in technologies—the ideas on how to use them stay the same. There are more and more solutions to the same kinds of problems:

text editors
CSS pre-processors
find-in-files tools
screen scraping tools
… many more …

In this blog post I’d like to tackle the problem from yet another perspective. Instead of resolving to “new and cool” libraries and languages (grep implemented in X language)—I’d like to use what’s out there already in terms of tooling to build a nice search-in-files tool for myself.

Search in files tools

It seems that for many people it’s very important to have a “search in files” tool that they really like. Some of the nice work we’ve seen so far include:

These are certainly very nice. As the goal of this post is to build something out of the tooling found in any minimal Unix-like installation—they won’t work though. They either need to be compiled or require Perl to be installed which isn’t everywhere (e. g. FreeBSD on default—though obviously available via the ports).

What I really need from the tool

I do understand that for some developers, waiting 100 ms longer for the search results might be too long. I’m not like that though. Personally, all I care about when searching is how the results are being presented. I also like to have the consistency of using the same approach between many machines I work on. We’re often working on remote machines at End Point. The need to install e.g Rust compiler just to get the ripgrep tool is too time consuming and hence doesn’t contribute to getting things done faster. Same goes for e. g the_silver_searcher which needs to be compiled too. What options do I have then?

Using good old Unix tools

The “find in files” functionality is covered fully by the Unix grep tool. It allows searching for a given substring but also “Regex” matches. The output can not only contain only the lines with matches, but also the lines before and after to give some context. The tool can provide line numbers and also search recursively within directories.

While I’m not into speeding it up, I’d certainly love to play with its output because I do care about my brain’s ability to parse text and hence: be more productive.

The usual output of grep:

$ # searching inside of the ripgrep repo sources:
$ egrep -nR Option src
(...)
src/search_stream.rs:46:    fn cause(&self) -> Option<&StdError> {
src/search_stream.rs:64:    opts: Options,
src/search_stream.rs:71:    line_count: Option<u64>,
src/search_stream.rs:78:/// Options for configuring search.
src/search_stream.rs:80:pub struct Options {
src/search_stream.rs:89:    pub max_count: Option<u64>,
src/search_stream.rs:94:impl Default for Options {
src/search_stream.rs:95:    fn default() -> Options {
src/search_stream.rs:96:        Options {
src/search_stream.rs:113:impl Options {
src/search_stream.rs:160:            opts: Options::default(),
src/search_stream.rs:236:    pub fn max_count(mut self, count: Option<u64>) -> Self {
src/search_stream.rs:674:    pub fn next(&mut self, buf: &[u8]) -> Option<(usize, usize)> {
src/worker.rs:24:    opts: Options,
src/worker.rs:28:struct Options {
src/worker.rs:38:    max_count: Option<u64>,
src/worker.rs:44:impl Default for Options {
src/worker.rs:45:    fn default() -> Options {
src/worker.rs:46:        Options {
src/worker.rs:72:            opts: Options::default(),
src/worker.rs:148:    pub fn max_count(mut self, count: Option<u64>) -> Self {
src/worker.rs:186:    opts: Options,
(...)

What my eyes would like to see is more like the following:

$ mygrep Option src
(...)
src/search_stream.rs:
 46        fn cause(&self) -> Option<&StdError> {
 ⁞
 64        opts: Options,
 ⁞
 71        line_count: Option<u64>,
 ⁞
 78    /// Options for configuring search.
 ⁞
 80    pub struct Options {
 ⁞
 89        pub max_count: Option<u64>,
 ⁞
 94    impl Default for Options {
 95        fn default() -> Options {
 96            Options {
 ⁞
 113   impl Options {
 ⁞
 160               opts: Options::default(),
 ⁞
 236       pub fn max_count(mut self, count: Option<u64>) -> Self {
 ⁞
 674       pub fn next(&mut self, buf: &[u8]) -> Option<(usize, usize)> {

src/worker.rs:
 24        opts: Options,
 ⁞
 28    struct Options {
 ⁞
 38        max_count: Option<u64>,
 ⁞
 44    impl Default for Options {
 45        fn default() -> Options {
 46            Options {
 ⁞
 72                opts: Options::default(),
 ⁞
 148       pub fn max_count(mut self, count: Option<u64>) -> Self {
 ⁞
 186       opts: Options,
(...)

Fortunately, even the tiniest of Unix like system installation already has all we need to make it happen without the need to install anything else. Let’s take a look at how we can modify the output of grep with awk to achieve what we need.

Piping into awk

Awk has been in Unix systems for many years—it’s older than me! It is a programming language interpreter designed specifically to work with text. In Unix, we can use pipes to direct output of one program to be the standard input of another in the following way:

$ oneapp | secondapp

The idea with our searching tool is to use what we already have and pipe it between the programs to format the output as we’d like:

$ egrep -nR Option src | awk -f script.awk

Notice that we used egrep when in this simple case we didn’t need to. It was sufficient to use fgrep or just grep.

Very quick introduction to coding with Awk

Awk is one of the forefathers of languages like Perl and Ruby. In fact some of the ideas I’ll show you here exist in them as well.

The structure of awk programs can be summarized as follows:

BEGIN {
  # init code goes here
}

# "body" of the script follows:

/pattern-1/ {
  # what to do with the line matching the pattern?
}

/pattern-n/ {
  # ...
}

END {
  # finalizing
}

The interpreter provides default versions for all three parts: a “no-op” for BEGIN and END and “print each line unmodified” for the “body” of the script.

Each line is being exploded into columns based on the “separator” which by default is any number of consecutive white characters. One can change it via the -F switch or by assigning the FS variable inside the BEGIN area. We’ll do just that in our example.

The “columns” that lines are being exploded into can be accessed via the special variables:

$0 # the whole line
$1 # first column
$2 # second column
# etc

The FS variable can contain a pattern too. So for example if we’d have a file with the following contents:

One | Two | Three | Four
Eins | Zwei | Drei | Vier
One | Zwei | Three | Vier

The following assignment would make Awk explode lines into proper columns:

BEGIN {
  FS="|"
}

# the ~ operator gives true if left side matches
# the regex denoted by the right side:
$1 ~ "One" {
  print $2
}

Running the following script would result with:

$ cat file.txt | awk -f script.awk
Two
Zwei

Simple Awk coding to format the search results

Armed with this simple knowledge, we can tackle the problem we stated in the earlier part of this post:

BEGIN {
  # the output of grep in the simple case
  # contains:
  # <file-name>:<line-number>:<file-fragment>
  # let's capture these parts into columns:
  FS=":"

  # we are going to need to "remember" if the <file-name>
  # changes to print it's name and to do that only
  # once per file:
  file=""

  # we'll be printing line numbers too; the non-consecutive
  # ones will be marked with the special line with vertical
  # dots; let's have a variable to keep track of the last
  # line number:
  ln=0

  # we also need to know we've just encountered a new file
  # not to print these vertical dots in such case:
  filestarted=0
}

# let's process every line except the ones grep prints to
# say if some binary file matched the predicate:
!/(--|Binary)/ {

  # remember: $1 is the first column which in our case is
  # the <file-name> part; The file variable is used to
  # store the file name recently processed; if the ones
  # don't match up - then we know we encountered a new
  # file name:
  if($1 != file && $1 != "")
  {
    file=$1
    print "\n" $1 ":"
    ln = $2
    filestarted=0
  }

  # if the line number isn't greater than the last one by
  # one then we're dealing with the result from non-consecutive
  # line; let's mark it with vertical dots:
  if($2 > ln + 1 && filestarted != 0)
  {
    print "⁞"
  }

  # the substr function returns a substring of a given one
  # starting at a given index; we need to print out the
  # search result found in a file; here's a gotcha: the results
  # may contain the ':' character as well! simply printing
  # $3 could potentially left out some portions of it;
  # this is why we're using the whole line, cutting off the
  # part we know for sure we don't need:
  out=substr($0, length($1 ":" $2 ": "))

  # let's deal with only the lines that make sense:
  if($2 >= ln && $2 != "")
  {
    # sprintf function matches the one found in C lang;
    # here we're making sure the line numbers are properly
    # spaced:
    linum=sprintf("%-4s", $2)

    # print <line-number> <found-string>
    print linum " " out

    # assign last line number for later use
    ln=$2

    # ensure that we know that we "started" current file:
    filestarted=1
  }
}

Notice that the “middle” part of the script (the one with the patterns and actions) gets ran in an implicit loop - once for each input line.

To use the above awk script you could wrap it up with the following shell script:

#!/bin/bash

egrep -nR $@ | awk -f script.awk

Here we’re very trivially (and somewhat naively) passing all the arguments passed to the script to egrep with the use of $@.

This of course is a simple solution. Some care needs to be applied when trying to make it work with A, B and C switches, it’s not difficult either though. All it takes is to e.g pipe it through sed (another great Unix tool - the “stream editor”) to replace the initial ‘-’ characters in the [filename]-[line-number] parts to match our assumptions of having “:“ as the separator in the awk script.

In praise of “what-already-works”

The simple script like shown above could easily be placed in your GitHub, BitBucket or GitLab account and fetched with curl on whichever machine you’re working on. With one call to curl and maybe another one to put the scripts somewhere in the local PATH you’d gain a productivity enhancing tool that doesn’t require anything else to work than what you already have.

I’ll keep learning “what we already have” to not fall too much into “what’s hot and new” unnecessarily.

Bash: loop over a list of (possibly non-existing) files using wildcards with nullglob (or failglob) option

2016-12-12T00:00:00+00:00

Let’s say you’re working in Bash, and you want to loop over a list of files, using wildcards.

The basic code is:

#!/bin/bash
for f in /path/to/files/*; do
  echo "Found file: $f"
done

Easy as that. However, there could be a problem with this code: if the wildcard does not expand to actual files (i.e. there’s no file under /path/to/files/ directory), $f will expand to the path string itself, and the for loop will still be executed one time with $f containing “/path/to/files/*”.

How to prevent this from happening? Nullglob is what you’re looking for.

Nullglob, quoting shopts man page, “allows filename patterns which match no files to expand to a null string, rather than themselves”.

Using shopt -s you can enable BASH optional behaviors, like Nullglob. Here’s the final code:

#!/bin/bash
shopt -s nullglob
for f in /path/to/files/*; do
  echo "Found file: $f"
done

Another interesting option you may want to check for, supported by Bash since version 3, is failglob.

With failglob enabled, quoting again, “patterns which fail to match filenames during filename expansion result in an expansion error”. Depending on what you need, that could even be a better behavior.

Wondering why nullglob it’s not the default behavior? Check this very good answer to the question.

Adding Bash Completion To a Python Script

2016-06-03T00:00:00+00:00

Bash has quite a nice feature, you can write a command in a console, and then press twice. This should should you possible options you can write for this command.

I will show how to integrate this mechanism into a custom python script with two types of arguments. What’s more, I want this to be totally generic. I don’t want to change it when I will change the options, or change config files.

This script accepts two types of arguments. One type contains mainly flags beginning with ‘–’, the other type is a host name taken from a bunch of chef scripts.

Let’s name this script show.py—it will show some information about the host. This way I can use it with:

show.py szymon

The szymon part is the name of my special host, and it is taken from one of our chef node definition files.

This script also takes huge number of arguments like:

show.py --cpu --memory --format=json

So we have two kinds of arguments: one is a simple string, one begins with –.

To implement the bash completion on double , first I wrote a simple python script, which is prints a huge list of all the node names:

#!/usr/bin/env python

from sys import argv
import os
import json

if __name__ == "__main__":
    pattern = ""
    if len(argv) == 2:
        pattern = argv[1]

    chef_dir = os.environ.get('CHEF_DIR', None)
    if not chef_dir:
        exit(0)
    node_dirs = [os.path.join(chef_dir, "nodes"),
                 os.path.join(chef_dir, "dev_nodes")]
    node_names = []

    for nodes_dir in node_dirs:
        for root, dirs, files in os.walk(nodes_dir):
            for f in files:
                try:
                    with open(os.path.join(root, f), 'r') as nf:
                        data = json.load(nf)
                        node_names.append(data['normal']['support_name'])
                except:
                    pass

    for name in node_names:
        print name

Another thing was to get a list of all the program options. We used the below one liner. It uses the help information shown by the script. So each time the script changed its options, and it is shown when used show.py –help, the tab completion will have show these new options.

$CHEF_DIR/repo_scripts/show.py --help | grep '  --' | awk '{print $1}'

The last step to make all this work was making a simple bash script, which uses the above python script, and the one liner. I placed this script in a file $CHEF_DIR/repo_scripts/show.bash-completion.

_show_complete()
{
    local cur prev opts node_names
    COMPREPLY=()
    cur="${COMP_WORDS[COMP_CWORD]}"
    prev="${COMP_WORDS[COMP_CWORD-1]}"
    opts=`$CHEF_DIR/repo_scripts/show.py --help | grep '  --' | awk '{print $1}'`
    node_names=`python $CHEF_DIR/repo_scripts/node_names.py`

    if [[ ${cur} == -* ]] ; then
        COMPREPLY=( $(compgen -W "${opts}" -- ${cur}) )
        return 0
    fi

    COMPREPLY=( $(compgen -W "${node_names}" -- ${cur}) )
}

complete -F _show_complete show.py

The last thing was to source this file, so I’ve added the below line in my ~/.bashrc.

source $CHEF_DIR/repo_scripts/show.bash-completion

And now pressing the twice in a console shows quite nice completion options:

$ show.py <tab><tab>
Display all 42 possibilities? (y or n)
... and here go all 42 node names ...
</tab></tab>

$ show.py h<tab><tab>
... and here go all node names beginning with 'h' ...
</tab></tab>

$ show.py --<tab><tab>
.. and here go all the options beginning with -- ...
</tab></tab>

Breaking Bash

2016-01-15T00:00:00+00:00

Recently I managed to break the bash shell in an interesting and puzzling way. The initial symptoms were very frustrating: a workflow process we use here (creating a development camp) failed for me, but for no one else. That was at least a clue that it was me, not the workflow process.

Eventually, I narrowed down the culprit to the “grep” command (and that was more through luck than steadfast Sherlock-like detective work).

$ grep foo bar

grep: foo: No such file or directory

Eh? grep is misparsing the arguments! How does that happen?

So I began to study my bash environment. Eventually I came up with this fascinating little typo:

export GREP_OPTIONS='-color=auto'

That’s supposed to be:

export GREP_OPTIONS='--color=auto'

but it got recorded in my .bashrc as a em-dash, not a double-dash. (My guess is that I cut-and-pasted this from a web page where someone over-helpfully “typeset” this command.)

Ironically, this typo is innocuous under Bash 3.x, but when you slot it into a Bash 4.x installation, all heck busts loose.

Top 15 Best Unix Command Line Tools

2015-11-04T00:00:00+00:00

Here are some of the Unix command line tools which we feel make our hands faster and lives easier. Let’s go through them in this post and make sure to leave a comment with your favourite!

1. Find the command that you are unaware of

In many situations we need to perform a command line operation but we might not know the right utility to run. The command (apropos) searches for the given keyword against its short description in the unix manual page and returns a list of commands that we may use to accomplish our need.

If you can not find the right utility, then Google is our friend :)

$ apropos "list dir"
$ man -k "find files"

2. Fix typos in our commands

It’s normal to make typographical errors when we type so fast. Consider a situation where we need to run a command with a long list of arguments and when executing it returns “command not found” and you noticed that you have made a typo on the executed command.

Now, we really do not want to retype the long list of arguments, instead use the following to simply just correct the typo command and execute ^typo_cmd^correct_cmd:

$ dc /tmp
$ ^dc^cd

The above will navigate to /tmp directory.

3. Bang and its Magic

Bang quite useful, when we want to play with the bash history commands. Bang helps by letting you execute commands in history easily when you need them:

!! — Execute the last executed command in the bash history
!* — Execute the command with all the arguments passed to the previous command
!ˆ — Get the first argument of the last executed command in the bash history
!$ — Get the last argument of the last executed command in the bash history
! — Execute a command which is in the specified number in bash history
!?keyword? — Execute a command from bash history for the first pattern match of the specified keyword
!-N — Execute the command that was Nth position from the last in bash history

$ ~/bin/lg-backup
 $ sudo !!

In the last part of the above example we didn’t realize that the lg-backup command had to be run with sudo. Now, Instead of typing the whole command again with sudo, we can just use sudo !! which will re-run the last executed command in bash history as sudo, which saves us lot of time.

4. Working with Incron

This incron configuration is almost like crontab setup, but the main difference is that incron monitors a directory for specific changes and triggers future actions as specified

Syntax: $directory $file_change_mask $command_or_action

/var/www/html/contents/ IN_CLOSE_WRITE,IN_CREATE,IN_DELETE /usr/bin/rsync –exclude ’*.tmp’ -a /home/ram/contents/ user@another_host:/home/ram/contents/
 /tmp IN_ALL_EVENTS logger "/tmp action for #file"

The above example shows triggering an rsync event whenever there is a change in /var/www/html/contents directory. In cases of immediate backup implementations this will be really helpful. Find more about incron here.

5. Double dash

There are situations where we end up in creating/deleting the directories whose name start with a symbol. These directories can not be removed by just using rm -rf or rmdir. So we need to use the “double dash” (–) to perform deletion of such directories:

$ rm -rf -- $symbol_dir

There are situations where you may want to create a few directory that starts with a symbol. You can just these create directories using double dash(–) and starting the directory name with a symbol.

$ mkdir -- $symbol_dir

6. Comma and Braces Operators

We can do lot with comma and braces to make our life easier when we are performing some operations, lets see few usages:

Rename and backup operations with comma & braces operator
Pattern matching with comma & braces operator
Rename and backup (prefixing name) operations on long file names

To backup httpd.conf to httpd.conf.bak:

$ cp httpd.conf{,.bak}

To revert the file from httpd.conf.bak to httpd.conf:

$ mv http.conf{.bak,}

To rename the file with prefix “old”:

$ cp exampleFile old-!#ˆ

7. Read only vim

As we all know, vim is a powerful command line editor. We can also use vim to view files in read only mode if you want to stick to vim:

$ vim -R filename

We can also use the view tool which is nothing but read only vim:

$ view filename

8. Push and Pop Directories

Sometimes when we are working with various directories and looking at the logs and executing scripts we find alot of our time is spent navigating the directory structure. If you think your directory navigations resembles a stack structure then use push and pop utilities which will save you lots of time

Push the directory using pushd
List the stack directories using the command dirs
Pop the directories using popd

This is mainly used in navigating between directories:

9. Copy text from Linux terminal (stdin) to the system clipboard

Install xclip and create the below alias:

$ alias pbcopy=’xclip -selection clipboard’
$ alias pbpaste=’xclip -selection clipboard -o’

We need to have the X window system running it to work. In Mac OS X, these pbcopy and pbpaste commands are readily available to you.

To Copy:

$ ls | pbcopy

To Paste:

$ pbpaste > lstxt.txt

10. TimeMachine like Incremental Backups in Linux using rsync –link-dest

This means that it will not recopy all of the files every single time a backup is performed. Instead, only the files that have been newly created or modified since the last backup will be copied. Unchanged files are hard linked from prevbackup to the destination directory.

$ rsync -a –link-dest=prevbackup src dst

11. To display the ASCII art of the Process tree

Showing your processes in a tree structure is very useful for confirming the relationship between every process running on your system. Here is an option which is available by default on most of the Linux systems:

$ ps -aux –forest

–forest is an argument to ps command, which displays ASCII art of process tree:

There are many commands available like pstree and htop to achieve the same thing.

12. Tree view of git commits

If you want to see git commits in a repo as tree view to understand the commit history better, the below option will be super helpful. This is available with the git installation and you do not need any additional packages.

$ git log –graph –oneline

13. Tee

Tee command is used to store and view (at the same time) the output of any other command.

(ie) At the same time it writes to the STDOUT, and to a file. It helps when you want to view the command output and at the time same time if you want to write it into a file or using pbcopy you can copy the output

$ crontab -l | tee crontab.backup.txt

The tee command is named after plumbing terminology for a T-shaped pipe splitter. This Unix command splits the output of a command, sending it to a file and to the terminal output. Thanks Jon for sharing this.

14. ncurses disk usage analyzer

Analysing disk usage with nurses interface, is fast and simple to use.

$ sudo apt-get install ncdu

15. hollywood

You all have seen the hacking scene on hollywood movies. Yes, there is a package which will let you create that for you.

$ sudo apt-add-repository ppa:hollywood/ppa
$ sudo apt-get update
$ sudo apt-get install hollywood

$ hollywood

Getting realtime output using Python Subprocess

2015-01-28T00:00:00+00:00

The Problem

When I launch a long running unix process within a python script, it waits until the process is finished, and only then do I get the complete output of my program. This is annoying if I’m running a process that takes a while to finish. And I want to capture the output and display it in the nice manner with clear formatting.

Using the subprocess and shlex library

Python has a “batteries included” philosophy. I have used 2 standard libraries to solve this problem.

import subprocess
import shlex

subprocess—Works with additional processes.
shlex—Lexical analysis of shell-style syntaxes.

subprocess.popen

To run a process and read all of its output, set the stdout value to PIPE and call communicate().

import subprocess
process = subprocess.Popen(['echo', '"Hello stdout"'], stdout=subprocess.PIPE)
stdout = process.communicate()[0]
print 'STDOUT:{}'.format(stdout)

The above script will wait for the process to complete and then it will display the output. So now we are going to read the stdout line by line and display it in the console untill it completes the process.

output = process.stdout.readline()

This will read a line from the stdout.

process.poll()

The poll() method will return

the exit code if the process is completed.
None if the process is still running.

while True:
        output = process.stdout.readline()
        if output == '' and process.poll() is not None:
            break
        if output:
            print output.strip()
    rc = process.poll()

The above will loop and keep on reading the stdout and check for the return code and displays the output in real time.

I had one more problem in parsing the shell commands to pass it to popen when I set the shell=False. Below is an example command:

rsync -avzXH --delete --exclude=*.swp --exclude=**/drivers.ini /media/lgisos/lg.iso root@42-a:/isodevice

To split the string using shell-like syntax I have used shlex library’s split method.

Here is the final code looks like

def run_command(command):
    process = subprocess.Popen(shlex.split(command), stdout=subprocess.PIPE)
    while True:
        output = process.stdout.readline()
        if output == '' and process.poll() is not None:
            break
        if output:
            print output.strip()
    rc = process.poll()
    return rc

MySQL to PostgreSQL Migration Tips

2014-11-26T00:00:00+00:00

I recently was involved in a project to migrate a client’s existing application from MySQL to PostgreSQL, and I wanted to record some of my experiences in doing so in the hopes they would be useful for others.

Note that these issues should not be considered exhaustive, but were taken from my notes of issues encountered and/or things that we had to take into consideration in this migration process.

Convert the schema

The first step is to convert the equivalent schema in your PostgreSQL system, generated from the original MySQL.

We used mysqldump --compatible=postgresql --no-data to get a dump which matched PostgreSQL’s quoting rules. This file still required some manual editing to cleanup some of the issues, such as removing MySQL’s “Engine” specification after a CREATE TABLE statement, but this resulted in a script in which we were able to create a skeleton PostgreSQL database with the correct database objects, names, types, etc.

Some of the considerations here include the database collations/charset. MySQL supports multiple collations/charset per database; in this case we ended up storing everything in UTF-8, which matched the encoding of the PostgreSQL database, so there were no additional changes needed here; otherwise, it would have been necessary to note the original encoding of the individual tables and later convert that to UTF-8 in the next step.

We needed to make the following modifications for datatypes:

MySQL Datatype	PostgreSQL Datatype
tinyint	int
int(NN)	int
blob	bytea*
datetime	timestamp with timezone
int unsigned	int**
enum('1')	bool
longtext	text
varbinary(NN)	bytea

* Note: we ended up converting these specific fields to text, just given the data that was stored in these fields in actuality, which just goes to show you should review your data.

** Note: because PostgreSQL does not have unsigned numeric types, if this feature is an important part of your data model you can/should add a CHECK constraint to the column in question to check that the value is non-negative.

A few other syntactic changes; MySQL’s UNIQUE KEY in the CREATE TABLE statement needs to just be UNIQUE.

Some of the MySQL indexes were defined as FULLTEXT indexes as well, which was a keyword PostgreSQL did not recognize. We made note of these, then created just normal indexes for the time being, intending to review to what extent these actually needed full text search capabilities.

Some of the AUTO_INCREMENT fields did not get the DEFAULT value set correctly to a sequence, because those types just ended up as integers without being declared a serial field, so we used the following query to correct this:

-- cleanup missing autoincrement fields

WITH
datasource AS (
    SELECT
        k.table_name,
        k.column_name,
        atttypid::regtype,
        adsrc
    FROM
        information_schema.key_column_usage k
    JOIN
        pg_attribute
    ON
        attrelid = k.table_name :: regclass AND
        attname = k.column_name
    LEFT JOIN
        pg_attrdef
    ON
        adrelid = k.table_name :: regclass AND
        adnum = k.ordinal_position
    WHERE
        table_name IN (
            SELECT table_name::text FROM information_schema.key_column_usage WHERE constraint_name LIKE '%_pkey' GROUP BY table_name HAVING count(table_name) = 1
        ) AND
        adsrc IS NULL AND
        atttypid = 'integer' ::regtype
),
frags AS (
    SELECT
        quote_ident(table_name || '_' || column_name || '_seq') AS q_seqname,
        quote_ident(table_name) as q_table,
        quote_ident(column_name) as q_col
    FROM
        datasource
),
queries AS (
    SELECT
        'CREATE SEQUENCE ' || q_seqname || ';
' ||
        'ALTER TABLE ' || q_table || ' ALTER COLUMN ' || q_col || $$ SET DEFAULT nextval('$$ || q_seqname || $$');
    $$ ||
        $$SELECT setval('$$ || q_seqname || $$',(SELECT max($$ || q_col || ') FROM ' || q_table || '));
' AS query
    FROM frags
)
SELECT
    COALESCE(string_agg(query, E'\n'),$$SELECT 'No autoincrement fixes needed';$$) AS queries FROM queries
\gset

BEGIN;
:queries
COMMIT;

Basically the idea is that we look for all table with a defined integer primary key (hand-waving it it by using the _pkey suffix in the constraint name), but without a current default value, then generate the equivalent SQL to create a sequence and set that table’s default value to the nextval() for the sequence in question. We also generate SQL to scan that table and set that sequence value to the next appropriate value for the column in question. (Since this is for a migration and we know we’ll be the only user accessing these tables we can ignore MVCC.)

Another interesting thing about this script is that we utilize psql’s ability to store results in a variable, using the \gset command, then we subsequently execute this SQL by interpolating that corresponding variable in the same script.

Convert the data

The next step was to prepare the data load from a MySQL data-only dump. Using a similar dump recipe as for the initial import, we used: mysqldump --compatible=postgresql --no-create-info --extended-insert > data.sql to save the data in a dump file so we could iteratively tweak our approach to cleaning up the MySQL data.

Using our dump file, we attempted a fresh load into the new PostgreSQL database. This failed initially due to multiple issues, including ones of invalid character encoding and stricter datatype interpretations in PostgreSQL.

What we ended up doing was to create a filter script to handle all of the “fixup” issues needed here. This involved decoding the data and reencoding to ensure we were using proper UTF8, performing some context-sensitive datatype conversions, etc.

Additional schema modifications

As we were already using a filter script to process the data dump, we decided to take the opportunity to fixup some warts in the current table definitions. This included some fields which were varchar, but should have actually been numeric or integer; as this was a non-trivial schema (100 tables) we were able to use PostgreSQL’s system views to identify a list of columns which should should be numeric and were currently not.

Since this was an ecommerce application, we identified columns that were likely candidates for data type reassignment based on field names *count, *qty, *price, *num.

Once we identified the fields in question, I wrote a script to generate the appropriate ALTER TABLE statements to first drop the default, change the column type, then set the new default. This was done via a mapping between table/column name and desired output type.

Convert the application

The final (and needless to say most involved step) was to convert the actual application itself to work with PostgreSQL. Despite the fact that these databases both speak SQL, we had to come up for solutions for the following issues:

Quotation styles

MySQL is more lax with its quoting styles, so some of this migration involved hunting down differences in quoting styles. The codebase contained lots of double-quoted string literals, which PostgreSQL interprets as identifiers, as well as the difference in quoting of column names (backticks for MySQL, double-quotes for PostgreSQL). These had to be identified wherever they appeared and fixed to use a consistent quoting style.

Specific unsupported syntax differences:

INSERT ON DUPLICATE KEY

MySQL supports the INSERT ON DUPLICATE KEY syntax. Modifying these queries involved creating a special UPSERT-style function to support the different options in use in the code base. We isolated and categorized the uses of INSERT ON DUPLICATE KEY UPDATE into several categories: those which did a straight record replace and those which did some sort of modification. I wrote a utility script (detailed later in this article) which served to replicate the logic needed to handle this as the application would expect.

Upcoming versions of PostgreSQL are likely to incorporate an INSERT … ON CONFLICT UPDATE/IGNORE syntax, which would produce a more direct method of handling migration of these sorts of queries.

INSERT IGNORE

MySQL’s INSERT … IGNORE syntax allows you to insert a row and effectively ignore a primary key violation, assuming that the rest of the row is valid. You can handle this case via creating a similar UPSERT function as in the previous point. Again, this case will be easily resolved if PostgreSQL adopts the INSERT … ON CONFLICT IGNORE syntax.

REPLACE INTO

MySQL’s REPLACE INTO syntax effectively does a DELETE followed by an INSERT; it basically ensures that a specific version of a row exists for the given primary key value. We handle this case by just modifying these queries to do an unconditional DELETE for the Primary Key in question followed by the corresponding INSERT. We ensure these are done within a single transaction so the result is atomic.

INTERVAL syntax

Date interval syntax can be slightly different in MySQL; intervals may be unquoted in MySQL, but must be quoted in PostgreSQL. This project necessitated hunting down several instances to add quoting of specific literal INTERVAL instances.

Function considerations

last_insert_id()

Many times when you insert a records into a MySQL table, later references to this are found using the last_insert_id() SQL function. These sorts of queries need to be modified to utilize the equivalent functionality using PostgreSQL sequences, such as the currval() function.

GROUP_CONCAT()

MySQL has the GROUP_CONCAT function, which serves as a string “join” of sorts. We emulate this behavior in PostgreSQL by using the string_agg aggregate function with the delimiter of choice.

CONCAT_WS() — expected to be but not an issue; Postgres has this function

PostgreSQL has included a CONCAT_WS() function since PostgreSQL 9.1, so this was not an issue with the specific migration, but could still be an issue if you are migrating to an older version of PostgreSQL.

str_to_date()

This function does not exist directly in PostgreSQL, but can be simulated using to_date(). Note however that the format string argument differs between MySQL and PostgreSQL’s versions.

date_format()

MySQL has a date_format() function which transforms a date type to a string with a given format option. PostgreSQL has similar functionality using the to_char() function; the main difference here lies in the format string specifier.

DateDiff()

DateDiff() does not exist in PostgreSQL, this is handled by transforming the function call to the equivalent date manipulation operators using the subtraction (-) operator.

rand() to random()

This is more-or-less a simple function rename, as the equivalent functionality for returning a random float between 0.0 <= x <= 1.0 exists in PostgreSQL and MySQL, it’s just what the function name itself is. The other difference is that MySQL supports a scale argument so the random number for rand(N) will be returned between 0.0 <= x <= N, whereas you’d have to scale the result in PostgreSQL yourself, via random() * N.

IF() to CASE WHEN ELSE

MySQL has an IF() function which returns the second argument in the case the first argument evaluates to true otherwise returns the third argument. This can be trivially converted from IF(expression1, arg2, arg3) to the equilvalent PostgreSQL syntax: CASE WHEN expression1 THEN arg2 ELSE arg3.

IFNULL() to COALESCE()

MySQL has a function IFNULL() which returns the first argument if it is not NULL, otherwise it returns the second argument. This can effectively be replaced by the PostgreSQL COALESCE() function, which serves the same purpose.

split_part()

MySQL has a built-in function called split_part() which allows you to access a specific index of an array delimited by a string. PostgreSQL also has this function, however the split_part() function in MySQL allows the index to be negative, in which case this returns the part from the right-hand side.

in MySQL:

split_part('a banana boat', ' ', -1) => 'boat'

in PostgreSQL:

split_part('a banana boat', ' ', -1) => // ERROR: field position must be greater than zero

I fixed this issue by creating a custom PL/pgSQL function to handle this case. (In my specific case, all of the negative indexes were -1; i.e., the last element in the array, so I created a function to return only the substring occurring after the last instance of the delimiter.)

Performance considerations

You may need to revisit COUNT(*) queries

MySQL MyISAM tables have a very fast COUNT() calculation, owing to queries taking a table lock (which means MySQL can cache the COUNT() result itself, since there can be no concurrent updates), while PostgreSQL utilizes MVCC in order to calculate table counts which necessitates a full table scan to see which rows are visible to the specific calling snapshot, so this assumption may need to be revisited in order to calculate an equivalent performant query.

GROUP BY vs DISTINCT ON

MySQL is much more (ahem) flexible when it comes to GROUP BY/aggregate queries, allowing some columns to be excluded in a GROUP BY or an aggregate function. Making the equivalent query in PostgreSQL involves transforming the query from SELECT … GROUP BY (cols) to a SELECT DISTINCT ON (cols) … and providing an explicit sort order for the rows.

More notes

Don’t be afraid to script things; in fact, I would go so far as to suggest that everything you do should be scripted. This process was complicated and there were lots of moving parts to ensure moved in tandem. There were changes being made on the site itself concurrently, so we were doing testing against a dump of the original database at a specific point-in-time. Having everything scripted ensured that this process was repeatable and testable, and that we could get to a specific point in the process without having to remember anything I’d done off-the-cuff.

In addition to scripting the actual SQL/migrations, I found it helpful to script the solutions to various classifications of problems. I wrote some scripts which I used to create some of the various scaffolding/boilerplate for the tables involved. This included a script which would create an UPSERT function for a specific table given the table name, which was used when replacing the INSERT ON DUPLICATE KEY UPDATE functions. This generated script could then be tailored to handle more complex logic beyond a simple UPDATE. (One instance here is an INSERT ON DUPLICATE KEY UPDATE which increased the count of a specific field in the table instead of replacing the value.)

#!/usr/bin/env perl
# -*-cperl-*-

use strict;
use warnings;

use Data::Dumper;

my $table = shift or die "Usage: $0 <table>\n";
my @cols = @ARGV;

my $dbh = DBI->connect(...);

my @raw_cols = @{ $dbh->column_info(undef, 'public', $table, '%')->fetchall_arrayref({}) };
my %raw_cols = map { $_->{COLUMN_NAME} => $_ } @raw_cols;

die "Can't find table $table\n" unless @raw_cols;

my @missing_cols = grep { ! defined $raw_cols{$_} } @cols;

die "Referenced non-existing columns: @missing_cols\n" if @missing_cols;

my %is_pk;

unless (@cols) {
    @cols = map { $_->{COLUMN_NAME} } @raw_cols;
}

my @pk_cols = $dbh->primary_key(undef, 'public', $table);

@is_pk{@pk_cols} = (1)x@pk_cols;

my @data_cols = grep { ! $is_pk{$_} } @cols;

die "Missing PK cols from column list!\n" unless @pk_cols == grep { $is_pk{$_} } @cols;
die "No data columns!\n" unless @data_cols;

print <<EOF
CREATE OR REPLACE FUNCTION
    upsert_$table (@{[
    join ', ' => map {
        "_$_ $raw_cols{ $_ }->{pg_type}"
    } @cols
]})
RETURNS void
LANGUAGE plpgsql
AS \$EOSQL\$
BEGIN
    LOOP
        UPDATE $table SET @{[
    join ', ' => map { "$_ = _$_" } @data_cols
]} WHERE @{[
    join ' AND ' => map { "$_ = _$_" } @pk_cols
]};
        IF FOUND THEN
            RETURN;
        END IF;
        BEGIN
            INSERT INTO $table (@{[join ',' => @cols]}) VALUES (@{[join ',' => map { "_$_" } @cols]});
            RETURN;
        EXCEPTION WHEN unique_violation THEN
            -- Do nothing, and loop to try the UPDATE again.
        END;
    END LOOP;
END
\$EOSQL\$
;
EOF
;

This script created an upsert function from a given table to update all columns by default, also allowing you to create one with a different number of columns upserted.

I also wrote scripts which could handle/validate some of the column datatype changes. Since there were large numbers of columns which were changed, often multiple in the same table, I was able to have this script create a single ALTER TABLE statement with multiple ALTER COLUMN TYPE USING clauses, plus be able to specify the actual method that these column changes were to take place. These included several different approaches, depending on the target data type, but generally were to solve cases where there were fairly legitimate data that was not picked up by PostgreSQL’s input parsers. These included how to interpret blank fields as integers (in some cases we wanted it to be 0, in others we wanted it to be NULL), weird numeric formatting (leaving off numbers before or after the decimal point), etc.

We had to fix up in several locations missing defaults for AUTO_INCREMENT columns. The tables were created with the proper datatype, however we had to find tables which matched a specific naming convention and create/associate a sequence/serial column, set the proper default here, etc. (This was detailed above.)

There was a fair amount of iteration and customization in this process, as there was a fair amount of data which was not of the expected format. The process was iterative, and generally involved attempting to alter the table from within a transaction and finding the next datum which the conversion to the expected type did not work. This would result in a modification of the USING clause of the ALTER TABLE ALTER COLUMN TYPE to accommodate some of the specific issues.

In several cases, there were only a couple records which had bad/bunko data, so I included explicit UPDATE statements to update those data values via primary key. While this felt a bit “impure”, it was a quick and preferred solution to the issue of a few specific records which did not fit general rules.

Python Subprocess Wrapping with sh

2014-07-24T00:00:00+00:00

When working with shell scripts written in bash/csh/etc. one of the primary tools you have to rely on is a simple method of piping output and input from subprocesses called by the script to create complex logic to accomplish the goal of the script. When working with python, this same method of calling subprocesses to redirect the input/output is available, but the overhead of using this method in python would be so cumbersome as to make python a less desirable scripting language. In effect you were implementing large parts of the I/O facilities, and potentially even writing replacements for the existing shell utilities that would perform the same work. Recently, python developers attempted to solve this problem, by updating an existing python subprocess wrapper library called pbs, into an easier to use library called sh.

Sh can be installed using pip, and the author has posted some documentation for the library here: http://amoffat.github.io/sh/

Using the sh library

After installing the library into your version of python, there will be two ways to call any existing shell command available to the system, firstly you can import the command as though it was itself a python library:

from sh import hostname
print(hostname())

In addition, you can also call the command directly by just referencing the sh namespace prior to the command name:

import sh
print(sh.hostname())

When running this command on my linux workstation (hostname atlas) it will return the expected results:

Python 2.7.6 (default, Mar 22 2014, 22:59:56)
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sh
>>> print(sh.hostname())
atlas

However at this point, we are merely replacing a single shell command which prints output to the screen, the real benefit of the shell scripts was that you could chain together commands in order to create complex logic to help you do work.

Advanced Gymnastics

A common use of shell scripts is to provide administrators the ability to quickly filter log file output and to potentially search for specific conditions within those logs, to alert in the event that an application starts throwing errors. With python piping in sh we can create a simple log watcher, which would be capable of calling anything we desire in python when the log file contains any of the conditions we are looking for.

To pipe together commands using the sh library, you would encapsulate each command in series to create a similar syntax to bash piping:

>>> print(sh.wc(sh.ls("-l", "/etc"), "-l"))
199

This command would have been equivalent to the bash pipe of “ls -l /etc | wc -l” indicating that the long listing of /etc on my workstation contained 199 lines of output. Each piped command is encapsulated inside the parenthesis of the command the precedes it.

For our log listener we will use the tail command along with a python iterator to watch for a potential error condition, which I will represent with the string “ERROR”:

>>> for line in sh.tail("-f", "/tmp/test_log", _iter=True):
...     if "ERROR" in line:
...         print line

In this example, once executed, python will call the tail command to follow a particular log file. It will iterate over each line of output produced by tail and if any of the lines contain the string we are watching for python will print that line to standard output. At this point, this would be similar to using the tail command and piping the output to a string search command, like grep. However, you could replace the third line of the python with a more complex action, emailing the error condition out to a developer or administrator for review, or perhaps initiating a procedure to recover from the error automatically.

Conclusions

In this manner with just a few lines of python, much like with bash, one could create a relatively complex process without recreating all the shell commands which already perform this work, or create a convoluted wrapping process of passing output from command to command. This combination of the existing shell commands and the power of python; you get all the functions available to any python environment, with the ease of using the shell commands to do some of the work. In the future I will definitely be using this python library for my own shell scripting needs, as I have generally preferred the syntax and ease of use of python over that of bash, but now I will be able to enjoy both at the same time.

Version differences via GitHub from the command line

2014-07-09T00:00:00+00:00

Photo by Benson Kua

I work with a lot of open source projects, and I use the command-line for almost everything. It often happens that I need to examine a file from a project, and thanks to bash, Github, and curl, I can do so easily, without even needing to have the repo handy. One of the things I do sometimes is compare a file across versions to see what has changed. For example, I needed to see what changes were made between versions 1.22 and 1.23 to the file includes/UserMailer.php which is part of the MediaWiki project. For this trick to work, the project must be on Github, and must label their versions in a consistent manner, either via git branches or git tags.

MediaWiki exists on Github as wikimedia/mediawiki-core. The MediaWiki project tags all of their releases in the format X.Y.Z, so in this example we can use the git tags 1.22.0 and 1.23.0. Github is very nice because you can view a specific file at a certain commit (aka a tag), and even grab it over the web as a plain text file. The format is:

https://raw.githubusercontent.com/PROJECTNAME/BRANCH-OR-TAG/FILE

Note that you can use a tag OR a branch! So to compare these two files, we can use one of these pairs:

https://raw.githubusercontent.com/wikimedia/mediawiki-core/REL1_21/includes/UserMailer.php
https://raw.githubusercontent.com/wikimedia/mediawiki-core/REL1_22/includes/UserMailer.php

https://raw.githubusercontent.com/wikimedia/mediawiki-core/1.21.0/includes/UserMailer.php
https://raw.githubusercontent.com/wikimedia/mediawiki-core/1.22.0/includes/UserMailer.php

All that is left is to treat git as a web service and compare the two files at the command line ourselves. The program curl is a great tool for downloading the files, as it dumps to stdout by default. We will add a -s flag (for “silent”) to prevent it from showing the progress meter as it usually does. The last bit of the puzzle is to use <(), bash’s process substitution feature, to trick diff into comparing the curl outputs as if they were files. So our final command is:

diff <(curl -s https://raw.githubusercontent.com/wikimedia/mediawiki-core/1.21.0/includes/UserMailer.php) \
<(curl -s https://raw.githubusercontent.com/wikimedia/mediawiki-core/1.22.0/includes/UserMailer.php) \
| more

Voila! A quick and simple glance at what changed between those two tags. This should work for any project on Github. You can also replace the branch or tag with the word “master” to see the current version. For example, the PostgreSQL project lives on github as postgres/postgres. They use the format RELX_Y_Z in their tags. To see what has changed since release 9.3.4 in the psql help file (as a context diff), run:

diff -c <(curl -s https://raw.githubusercontent.com/postgres/postgres/REL9_3_4/src/bin/psql/help.c) \
<(curl -s https://raw.githubusercontent.com/postgres/postgres/master/src/bin/psql/help.c)

You are not limited to diff, of course. For a final example, let’s see how many times Tom Lane is mentioned in the version 9 release notes:

for i in {0,1,2,3,4}
do grep -Fc 'Tom Lane' \
<(curl -s https://raw.githubusercontent.com/postgres/postgres/master/doc/src/sgml/release-9.$i.sgml)
done
272
206
174
115
16

The last number is so low relative to the rest because 9.4 is still under development. Rest assured Tom’s contributions have not slowed down! :) Thanks to Github for providing such a useful service for so many open source projects, and for providing the raw text to allow useful hacks like this.

Changing Postgres pg_dump warnings into errors with sed

2013-10-28T00:00:00+00:00

Turning off warnings when loading a PostgreSQL data dump can be a trickier task than it may first appear. One of the goals of a program I was working on was to automatically load the schema of a production database into a development system. The program needed to stay as quiet as possible - the users running it did not need to be distracted by annoying messages generated by Postgres as the schema was loaded.

The problem occurred when a text-based dump of the database (created via pg_dump with the –schema-only flag) was loaded into an existing database with the help of the the psql program. To load the file “test.sql” into the database wilber, one usually does:

$ psql wilber -U postgres -f test.sql

There are two general classes of “noise” that are generated by such an action. I’ll show how to make both kinds quiet.

The first problem is that psql by default is very verbose and every single command gives an echoing confirmation of what just occurred. Thus, a typical schema load outputs a lot of things like this:

$ psql wilber -U postgres -f test.sql
SET
SET
SET
SET
SET
CREATE EXTENSION
COMMENT
SET
SET
SET
CREATE TABLE
ALTER TABLE
ALTER TABLE
CREATE SEQUENCE

These types of messages are easy to turn off. Simply add the –quiet flag when calling psql, usually abbreviated to just -q, like so:

$ psql wilber -q -U postgres -f test.sql

The other type of unwanted output that can appear are the various messages from Postgres itself. These have different levels of severity: which ones are shown to you are determined by the client_min_messages setting. From least important to most important, the levels are:

DEBUG
LOG
NOTICE
WARNING
ERROR
FATAL
PANIC

The output I was seeing looked like this:

psql:test.sql:22716: WARNING:  => is deprecated as an operator name
DETAIL:  This name may be disallowed altogether in future versions of PostgreSQL.

Not something I wanted users to see every time they ran the script! The solution was to set client_min_messages to something higher than WARNING. In this case, ERROR is a good choice. All warning-level notices will be suppressed. The SQL would look like this:

SET client_min_messages = ERROR;

However, because we are loading a file, there is no easy way to enter in SQL commands before the file is loaded. Luckily, there is a very-poorly-documented trick that can apply some settings before a program (such as psql) is run. Just set the PGOPTIONS environment variable. It allows you to pass certain options to psql and other programs. For the case at hand, you would do this:

$ PGOPTIONS='--client-min-messages=error' psql wilber -q -U postgres -f test.sql

This will ensure that client_min_messages is set to ERROR before psql starts to load the file. If you are using a Perl script (as I was), you can set the environment variable before making a system call:

$ENV{PGOPTIONS} = '--client-min-messages=error';
$COM = 'psql wilber -q -U postgres -f test.sql';
system $COM;

However, if your file is from a pg_dump, you will find that this trick does not work! Why? When pg_dump creates a file, it adds some SET commands to the top of it. One of those is client_min_messages, which will clobber whatever you have set it to. D’oh! Here’s what pg_dump currently sticks at the top of its output:

$ pg_dump --schema-only | head -20 | grep -n '^SET'
5:SET statement_timeout = 0;
6:SET client_encoding = 'UTF8';
7:SET standard_conforming_strings = on;
8:SET check_function_bodies = false;
9:SET client_min_messages = warning;

What to do? The schema file was very large, so editing it was not an option—and certainly not for this automated task. The answer was to modify the large schema text file before loading it. While perl is my goto tool for most things, this looked the perfect job for the sed program. So, in my script, I simply added this right before psql is invoked:

  $COM = "sed --in-place '1,/client_min_messages/ {s/warning/ERROR/}' test.sql";
  system $COM;

The short explanation is that it replaces the ‘warning’ on line 9 of the file with the word ‘error’. This prevented any of the annoying warnings from showing up, and made the script nice and quietly happy again.

The long explanation about how it works is the rest of this post. :)

The sed program is a “stream editor” designed to quickly transform text. The man page is not very useful: I recommend learning more about sed via an online tutorial. This one is my favorite, not least because of the “modem noise” joke (kids, look that one up).

First we instruct sed, via the –in-place flag to make the changes directly to the file itself, rather than sending output to stdout. This is usually abbreviated to simply -i, but I consider the longer form more polite, as it’s a little less obtuse to someone not familiar with the flags. Consider it a cheap way to comment your code.

After that, in single quotes, is the actual instructions we pass to sed. Finally, we give it the file to perform the work on. Again, the instructions to sed were:

'1,/client_min_messages/ {s/warning/ERROR/}'

This has two sections: what to do, and where to do it. By default, sed operates on every line of the file. As we only want sed to change the first occurrence of client_min_messages, we tell it to limit its actions:

1,/client_min_messages/

You can instruct sed to operate on a range of lines by using the X,Y format, aka start,stop. A lone number represents a line number in the file; in this case we want to start at the first line of the file. You can also provide a regular expression. In this case, we want to stop processing when we find a line that matches the regular expression /client_min_messages/. Note that these limits are inclusive, so that we still process the matched line.

The next section tells sed what to do:

' {s/warning/ERROR/}'

Note that both the extra space and the braces are not needed, but are there simply to make things easier to read. You can also change the delimiter, which can arguably help as well; here is an equivalent version:

'1,/client_min_messages/ {s:warning:ERROR:}'

We are telling sed to make a simple substitution of the word warning with the word ERROR. I used uppercase here to make it more obvious to anyone viewing the file that a change had been made (Postgres does not care what the case is).

Normally, this might be considered a fairly sloppy regex, as it doesn’t match very strictly—any lines up to and including the client_min_messages will have the word warning replaced. However, because we can be confident that the pg_dump output is going to remain very standard, we can let it slide. A more proper solution might be:

'1,/client_min_messages/ {s:client_min_messages = warning:client_min_messages = ERROR:}'

Why didn’t I simply replace the regex and be done, like this?:

's/client_min_messages = warning/client_min_messages = ERROR/'

Two reasons. First, by setting a range, we ensure that we only make the change at the top of the file, and not anywhere else where it may be set (not by pg_dump itself, but perhaps as the text inside a function. You never know). This is also more efficient, as sed doesn’t need to scan the lines for a potential replacement after it hits the first occurrence at line 9.

The second reason is that we can now safely run this against the file more than once, and not worry about a bad replacement or about losing efficiency. If we search for the ‘warning’ version of the regex, then the second time through we may end up scanning the entire file (and any substitutions we do make will be in the wrong place).

To recap, the winning lines in Perl were:

$COM = "sed --in-place '1,/client_min_messages/ {s/warning/error/}' test.sql";
system $COM;

Once we have this, we no longer need the PGOPTIONS setting. Which was a little dangerous as a global $ENV inside the perl script anyways, as the setting persists even after the first system call! So sed has saved the day, or at least made the users of this program happier!

Making use of a Unix Pipe

2013-06-13T00:00:00+00:00

Developing in a Unix-based environment has many wonderful advantages. Thanks to Gary Bernhardt of DestroyAllSoftware Screencasts, I’ve recently discovered a new use for the Unix pipe. A pipe in Unix does exactly what you might think it would do by its name. Send something in one side and watch it come out the other. If you’ve done much in the shell, you’ve probably used pipes before where you’ve probably piped some output from one command to another. Here’s an example:

$ cat foo.txt | grep bar

This command simply says take the output of cat and sends it to the input of grep. Pipes used in this way can yield very powerful commands in the shell.

There is another pipe in Unix and this is a named pipe. A named pipe, or a FIFO (First In, First Out), works similarly to command line pipe. You put stuff in one end and it comes out the other. To create a named pipe, you use the mkfifo command.

$ mkfifo my_fifo
$ ls -l
...
prw-rw-r--  1 mikefarmer mikefarmer      0 Jun  5 21:22 my_fifo

Notice the “p” at the beginning of the file list. The “p” designates this file as a named pipe to the system. To try out our pipe, we will, in one terminal, listen for anything coming out of the pipe. Then in another terminal we will send some text to the pipe. To listen to the pipe we will be using cat to just display whatever comes into the pipe.

$ cat my_fifo

Notice that when this runs, it blocks while it waits for something to come through the pipe. Now let’s push some text through the pipe. In another terminal window, I’ll just echo some text to the pipe.

$ echo "hello world" > my_fifo

As soon as I press enter on this command, I see that the other terminal outputs “hello world” and then exits.

Now that we have a basic understanding of how the pipe works, we can set it up to run some commands. I’m going to throw my listener into a shell script that will create our pipe and start listening to and executing anything that comes out of it.

  # setup_listener.sh
  if [ ! -p commands ]; then
    mkfifo commands
  fi

  while true; do
    sh -c "clear && $(cat commands)"
  done

The first three lines create the pipe. The last three lines setup the listener. Remember how the process ended after the first command was sent? Well, putting this into a loop allows us to call cat repeatedly. I also added a clear call to clear my screen between each command. Now to test it out:

$ sh ./setup_listener.rb

In another terminal:

$ echo 'ls -l' > commands

You’ll notice that the command clears the screen in the other terminal and displays the output of the command ls -l. Nice!

Now let’s put this to some practical use. I have a Rails application that has some Minitest tests and a small script that I’ve put together to run all the tests. Here’s the content of of my test runner:

# run_all_tests.rb

#!/usr/bin/env ruby
$:<<'test'

files = Dir.glob('test/**/*_test.rb')
files.each{|file| require file.sub(/^test\/|.rb$/,'')}

The script is a simple Ruby script that adds the test directory to the LOAD_PATH and then just requires all the files in the test directory that start with “test”. I make this script executable using a chmod +x command. Then to run it, I just call

$ ./run_all_tests.rb

Simple. To run individual tests, I just run:

$ minitest test/test_foo.rb

With these two commands in mind, I can now put together everything I need for my pipe. First I’m going to vertically split my screen (You can use tmux, or whatever tool you’d like. I like iTerm’s simple split screen for this.) In my terminal on the right, I’m going to startup my listener just like I did above. In the terminal on the right, I’m going to start up vim and bring up my test file. To execute my test, I’ll just use vim’s :! command to execute the test command in the shell using the % as a placeholder for the current file name in my active buffer.

:!echo "minitest %" > commands

If I’ve done everything right, my test will run on the terminal on the right and then wait for my next command. W00T! Now I’m going to run all my tests:

:!echo "./run_all_tests.rb" > commands

Immediately upon pressing enter, all my tests are running on the right pane! Hooray!

To make things a little easier in vim, I setup some key mappings for running the tests.

:nmap <leader>g :w\|:silent !echo "minitest %" > commands<cr>
:nmap <leader>G :w\|:silent !echo "./run_all_tests.rb" > commands<cr>

Now I don’t necessarily want these shortcuts all the time so I’m going to add these shortcuts to a file called setup_test_shortcuts.vim. Then to activate them I just run :source setup_test_shortcuts.vim. Now I have a simple shortcut in vim for running my tests!

If you are using zeus then you will need to modify your shortcuts to look like this:

nmap <leader>g :w\|:silent !echo "zeus test %" > commands<cr>
nmap <leader>G :w\|:silent !echo "zeus test ./run_all_tests.rb" > commands<cr>

Using Unix to help me in my workflow always brings a big smile to my face as I see the gains in productivity. Simple Unix concepts continue to blow my mind at their practicality and underlying simplicity.

Adventures with using Ruby 2.0 and libreadline

2013-05-17T00:00:00+00:00

I was asked to develop a prototype app for one of our clients lately. The basis for this app was an old Rails app:

Rails 3.2.8
RailsAdmin
MySQL
rbenv + ruby-build

I wanted to upgrade the stack to work with latest toys all cool kids are so thrilled about. I also didn’t have Rails console facility at my disposal since the Ruby version installed on the development machine hadn’t been compiled against libreadline.

Not having root or sudo access on the machine I embarked on a sligthly hacky journey to make myself a better working environment.

Ruby 2.0

After reading Mike Farmer’s blog post about Ruby 2.0 and tons of other material about it on the Internet, I wanted to get a feeling of how faster & greater the new Ruby is. It’s always great also to stay up-to-date with latest technologies. It’s great for me as a developer, and more importantly—it’s great for our clients.

Importance of libreadline in development with Ruby

To be productive developing any Rails-based application, we have to have Rails-console available at any moment. It serves a multitude of purposes. It’s also a great scratch-pad when developing methods.

While you don’t need your Ruby to support libreadline for basic uses of irb, you need it when using with Rails.

Installing Ruby 2.0.0 with rbenv (ruby-build)

If you’ve installed ruby-build some time ago, chances are that you need to update it in order to be able to install latest build of Ruby 2.0.0

To do it:

cd ~/.rbenv/plugins/ruby-build
git pull

And you should be able now to have available latest Ruby build to install:

rbenv install 2.0.0-p195

If you want to install Ruby compiled with support for libreadline, you have to have it installed in your system before compiling the build with rbenv install.

If you have access to root or sudo on your system, the easiest way is to e. g:

on Debian-related Linuxes:

apt-get install libreadline-dev

or on Fedora:

yum install readline-devel

Installing libreadline from sources

In my case—I had to download sources and compile them myself. Luckily the system had all needed essential packages installed for building it.

wget "ftp://ftp.cwru.edu/pub/bash/readline-6.2.tar.gz"
tar xvf readline-6.2.tar.gz
cd readline-6.2
./configure --prefix=/home/kamil/libs
make
make install

I had to specify –prefix option pointing at the path where I wanted the libreadline library to be installed after compilation.

Then, I was able to actually build Ruby with readline support “on”:

CONFIGURE_OPTS="--with-readline-dir=/home/kamil/libs" rbenv install 2.0.0-p195

Notice: I was making myself a development environment and compiling from sources was my last resort. It is not a good practice for production environments.

Last thing I needed to do was to get rb-readline working with the project I was working on.

It turnes out that latest rb-readline doesn’t play well with latest Ruby. Also, when using Ruby 2.0.0 one have to explicitely specify it in the Gemfile, or else it won’t be loaded for the console.

Gemfile:

gem 'rb-readline', '~> 0.4.2'

This still isn’t perfect

While this setup works, it won’t let you use arrow keys. The irb process crashes quickly after even first try to navigate through the text.

For some reason, after upgrading Ruby, the RailsAdmin stylesheets stopped working. I noticed that they are being served with comments which should be replaced by other stylesheets like:

/* ...
*= require_self
*= require_tree .
*/

I had to update Rails version in the Gemfile to have my admin back:

Gemfile:

gem 'rails', '3.2.13'

Console:

bundle

Last thing I wanted to do, was to try if I could upgrade Rails even further and have a working Rails4 setup. This was impossible unfortunately since RailsAdmin isn’t yet compatible with it as stated here.

I conclude that latest Ruby is quite usable right now. If you don’t mind the quirks with the readline—you’re pretty safe to upgrade. This assumes though that your app doesn’t use any incompatible elements.

The main Ruby site describes them like so:

There are five notable incompatibilities we know of:

The default encoding for ruby scripts is now UTF-8 [#6679]. Some people report that it affects existing programs, such as some benchmark programs becoming very slow [ruby-dev:46547].
Iconv was removed, which had already been deprecated when M17N was introduced in ruby 1.9. Use String#encode, etc. instead.
There is ABI breakage [ruby-core:48984]. We think that normal users can/should just reinstall extension libraries. You should be aware: DO NOT COPY .so OR .bundle FILES FROM 1.9.
#lines, #chars, #codepoints, #bytes now returns an Array instead of an Enumerator [#6670]. This change allows you to avoid the common idiom “lines.to_a”. Use #each_line, etc. to get an Enumerator.
Object#inspect does always return a string like #<ClassName:0x…> instead of delegating to #to_s. [#2152]
There are some comparatively small incompatibilities. [ruby-core:49119]

Camp tools

2013-01-14T00:00:00+00:00

Devcamps are such a big part of my everyday work that I can’t imagine life without them. Over the years, I developed some short-cuts in navigating camps that I also can’t live without: I share them below.

function camp_top() {
  if [ -n "$1" ]
  then
      cd ~/camp${1}
  elif [[ $(pwd) =~ 'camp' ]]
  then
      until [[ $(basename $(pwd)) =~ '^camp[[:digit:]]+' ]]
      do
          if [[ $(pwd) =~ 'camp' ]]
          then
              cd ..
          else
              break
          fi
      done
  fi
}
alias ct='camp_top; pwd'

function cat_root() {
  camp_top $*
  cd catalogs/* >/dev/null
}
alias cr='cat_root; pwd'

function pages_root() {
  cat_root $*
  cd pages >/dev/null
}
alias pr='pages_root; pwd'

function what_camp() {
  c=$( camp_top $* 2> /dev/null; basename $( pwd ))
  echo $c
}

(“cat_root” and “pages_root” are very Interchange-specific; you may find other short-cuts more useful in your particular camp.)

There’s nothing terribly ground-breaking here, but if bash is not your native shell-tongue, then you might find these useful.

What I do is to stash these somewhere like “$HOME/.bash_camps”, then alter my .bashrc:

# Source campy definitions
if [ -f ~/.bash_camps ]; then
 . ~/.bash_camps
fi

That’s all it takes. Have you a camp-y shell script, function, or alias? Please share in the comments!

Simple bash shell script for running batch MySQL jobs

2012-10-16T00:00:00+00:00

The other day I needed to run a simple mysql job to backup and delete some database records on a live server. Being a live server, it is important to make sure you aren’t asking the database to take on jobs that could potentially lock it up. Better to run a batch job. Running a batch is simple. You can call it right from the mysql console with:

source [path_to]/[the_batch_script].sql

But what if there are millions of records that need deleting? Bash shell script to the rescue.

Here is the idea of the SQL job that needed to get run a few times:

START TRANSACTION;

/* Find what you want to delete and put a LIMIT on your batch size */
CREATE TEMPORARY TABLE records_to_delete_temp SELECT id from `records` where ..... limit 1000;

/* Creating backup table to archive spam orders */
CREATE TABLE IF NOT EXISTS `records_backup` LIKE `records`;
INSERT INTO `records_backup` SELECT * from `records` where id in (select id from `records_to_delete_temp`);

/* Delete Dependents - If your records have foreign key dependencies, delete them first */
DELETE FROM `dependent_1` where record_id in (select id from `records_to_delete_temp`);
DELETE FROM `dependent_2` where record_id in (select id from `records_to_delete_temp`);

/* Delete the records */
DELETE FROM `records` where id in (select id from `records_to_delete_temp`);

/* Return count of remaining records - the where clause should be the same as the original select for the records to delete */
SELECT COUNT(*) from `records` where .... ;

COMMIT ;

Note:

SELECT COUNT(*)

This will return the remaining record count to the shell script.

And the shell script…

ret=$(mysql -u [user] -p[password] --skip-column-names [database_name]  < [the_batch_script].sql)

while [ $ret -gt 0 ]
do
  echo "rows left: $ret"
  sleep 3
  ret=$(mysql -u [user] -p[password] --skip-column-names [database_name]  < [the_batch_script].sql)
done

Notes:

--skip-column-names

This is the little nugget that gives you clean output from the mysql script. Without –skip-column-names you will have to get more creative with parsing the returned table.

-p[password]

Just a friendly reminder, no space after the -p option.

Should you really be running batches to remove millions of records from a live server with a shell script?

Just because you can doesn’t mean you should. Before pulling the trigger, back up, consider what could go wrong, have a plan in place to address the possible failure.

Find something fun for your hands to do while bash takes care of running the jobs. At the least, cross your fingers!

File test comparison table for shell, Perl, Ruby, and Python

2009-08-31T00:00:00+00:00

A few days ago, my co-worker Richard asked how in Python you would do the -x Bourne shell and Perl file test that checks whether a file is executable. This is (for me, at least) a really commonly used function but one I hadn’t needed to do yet in Python, so I looked it up.

That wasn’t so hard to find, but then I wondered about the other shell and Perl file tests that I use all the time. Finding equivalents for those was harder than I expected. A web search didn’t turn much up aside from language holy wars and limited answers, but I didn’t find any exhaustive list.

So I made my own. Below is a table comparing file test operators in the original Bourne shell-compatibles bash, ksh, and zsh; Perl’s expanded set; Ruby’s which was derived first from Perl; and equivalent Python code.

There are still some blanks where I didn’t find a good equivalent. Of course I’m sure it’s possible with enough custom logic to achieve the same end, but I have tried to stick with relatively simple formulations using built-in functions for now. I’ll be happy to fill in the blanks if any readers make suggestions.

Performance notes on avoiding multiple stats of the same file:

Starting with Perl 5.9.1, file tests can be “stacked” and will use a single stat for all tests, e.g. -f -x file. In older versions of Perl you can do -f file && -x _ instead.
Ruby’s File::Stat class can be used to cache a stat for multiple tests.
Python’s os.stat(file).st_mode can be stored and used for multiple tests.

Unless otherwise specified, these tests follow symbolic links and operate on the target of the link, rather than the link itself.

All tests return boolean true or false unless otherwise noted.

Test	bash/ksh/zsh	Perl	Ruby	Python
File is readable by effective uid/gid		-r 'file'	test ?r, 'file'	os.access('file', os.R_OK)
File.readable?('file')
File is writable by effective uid/gid		-w 'file'	test ?w, 'file'	os.access('file', os.W_OK)
File.writable?('file')
File is executable by effective uid/gid		-x 'file'	test ?x, 'file'	os.access('file', os.X_OK)
File.executable?('file')
File is owned by effective uid	-O file	-o 'file'	test ?o, 'file'	os.stat('file').st_uid == os.geteuid()
File.owned?('file')
File is owned by the effective gid	-G file	(stat('file'))[5] == $)	test ?G, 'file'	os.stat('file').st_gid == os.getegid()
File.grpowned?('file')
File is readable by real uid/gid	-r file	-R 'file'	test ?R, 'file'	os.access('file', os.R_OK)
File.readable_real?('file')
File is writable by real uid/gid	-w file	-W 'file'	test ?W, 'file'	os.access('file', os.W_OK)
File.writable_real?('file')
File is executable by real uid/gid	-x file	-X 'file'	test ?X, 'file'	os.access('file', os.X_OK)
File.executable_real?('file')
File is owned by real uid		-O 'file'	test ?O, 'file'	os.stat('file').st_uid == os.getuid()
File exists	-e file	-e 'file'	test ?e, 'file'	os.path.exists('file')
-a file	File.exist?('file')
File has zero size (is empty)	-f file -a ! -s file	-z 'file'	test ?z, 'file'	os.path.getsize('file') == 0
File.zero?('file')	os.stat('file').st_size == 0
File exists and has size greater than zero	-s file	-s 'file' (boolean and returns size in bytes)	test ?s, 'file' (boolean: returns nil if doesn't exist or has zero size, size of the file otherwise)	os.path.getsize('file') > 0
File.size?('file') (same)	os.stat('file').st_size > 0
File exists, return size in bytes		-s 'file'	File.size('file')	os.path.getsize('file')
os.stat('file').st_size
File is a plain file	-f file	-f 'file'	test ?f, 'file'	os.path.isfile('file')
File.file?('file')	stat.S_ISREG(os.stat('file').st_mode)
File is a directory	-d file	-d 'file'	test ?d, 'file'	os.path.isdir('file')
File.directory?('file')	stat.S_ISDIR(os.stat('file').st_mode)
File is a symbolic link	-h file	-l 'file'	test ?l, 'file'	os.path.islink('file')
-L file	File.symlink?('file')	stat.S_ISLNK(os.lstat('file').st_mode)
File is a named pipe (FIFO)	-p file	-p 'file' (can also be used on a filehandle)	test ?p, 'file'	stat.S_ISFIFO(os.stat('file').st_mode)
File.pipe?('file')
File is a socket	-S file	-S 'file'	test ?S, 'file'	stat.S_ISSOCK(os.stat('file').st_mode)
File.socket?('file')
File is a block special file	-b file	-b 'file'	test ?b, 'file'	stat.S_ISBLK(os.stat('file').st_mode)
File.blockdev?('file')
File is a character special file	-c file	-c 'file'	test ?c, 'file'	stat.S_ISCHR(os.stat('file').st_mode)
File.chardev?('file')
File type (returns string 'file', 'directory', 'characterSpecial', 'blockSpecial', 'fifo', 'link', 'socket', or 'unknown'			File.ftype('file')
Filehandle or descriptor is opened to a tty	-t fd	-t $fh	fd.isatty	os.isatty(fd)
fd.tty?
File has setuid bit set	-u file	-u 'file'	test ?u, 'file'	os.stat('file').st_mode & stat.S_ISUID
File.setuid?('file')
File has setgid bit set	-g file	-g 'file'	test ?g, 'file'	os.stat('file').st_mode & stat.S_ISGID
File.setgid?('file')
File has sticky bit set	-k file	-k 'file'	test ?k, 'file'	os.stat('file').st_mode & stat.S_ISVTX
File.sticky?('file')
File is an ASCII text file (heuristic guess)		-T 'file'
File is a "binary" file (opposite of -T)		-B 'file'
File modification time		(stat('file'))[9]	test ?M, 'file' (returns Time object)	os.stat('file').st_mtime
-M 'file' (script start time minus file modification time, in days)	File.mtime('file') (same)
File access time		(stat('file'))[8]	test ?A, 'file' (returns Time object)	os.stat('file').st_atime
-A 'file' (script start time minus file access time, in days)	File.atime('file') (same)
Inode change time (Unix)		(stat('file'))[10]	test ?C, 'file' (returns Time object)	os.stat('file').st_ctime
-C 'file' (script start time minus inode change time, in days)	File.ctime('file') (same)
File has been modified since it was last read	-N file
file1 is newer (according to modification date) than file2, or if file1 exists and file2 does not	file1 -nt file2	(stat('file1'))[9] > (stat('file2'))[9]	test ?>, 'file1', 'file2'	os.path.exists('file1') and (not os.path.exists('file2') or os.stat('file1').st_mtime > os.stat('file2').st_mtime)
file1 is older than file2, or if file2 exists and file1 does not	file1 -ot file2	(stat('file1'))[9] < (stat('file2'))[9]	test ?<, 'file1', 'file2'	os.path.exists('file2') and (not os.path.exists('file1') or os.stat('file1').st_mtime < os.stat('file2').st_mtime)
file1 and file2 refer to the same device and inode numbers	file1 -ef file2	join(':', (stat('file1'))[0,1]) eq join(':', (stat('file2'))[0,1])	test ?-, 'file1', 'file2'	os.path.samefile('file1', 'file2')
file1 and file2 have the same modification times		(stat('file1'))[9] == (stat('file2'))[9]	test ?=, 'file1', 'file2'	os.stat('file1').st_mtime == os.stat('file2').st_mtime

Complete details are in the manuals for each language:

bash: man bash and search for “CONDITIONAL EXPRESSIONS”
ksh: man pdksh and search for “test expression”
zsh: man zshmisc and search for “CONDITIONAL EXPRESSIONS”
Perl: perldoc -f -f, filetest pragma
Ruby: File class (summary version) and test() built-in, IO class docs
Python: os.access, os.stat, stat() results, os.path

Bash expansion techniques for a more efficient workflow

Shell parameter expansion

Example: batch converting images to WebP

Alternate method with parameter expansion search and replace

Example: counting file endings

History Expansion

Example: Replacing underscores with dashes in a filename

Other types of expansion

How to Analyze Application Logs and Extract Actionable Insights

A Note on Commands

1. Debugging Application Errors

1.1 Summarize Frequent Issues

1.2 Categorize Errors by Type

1.3 Find Problematic Scripts

2. Monitoring System Behavior

2.1 Track Problematic IPs

2.2 Spot High-Error Time Periods

2.3 Analyze IP–Referer Pairs

3. Enhancing Security

3.1 Watch Sensitive URLs

3.2 Pinpoint Critical Times

Conclusion

Practical Linux Command Line Tips for Productivity and Efficiency

1. Combining Commands with Pipes

Understanding How Pipes Work

Practical Examples: Real-World Use Cases for Combining Commands

Example 1: Extracting Information from Log Files

Example 2: Processing XML Files for Public Health Data

Example 3: Organizing Text Data into a CSV File

2. Mastering Shell Techniques

Understanding the Shell with ls

1. Pattern Matching for Filenames

2. Variables to Store Values

3. Redirection of Input and Output

4. Quoting and Escaping to Disable Shell Features

5. The Search Path for Locating Programs to Run

6. Saving Changes to Your Shell Environment

3. Efficient Command Recall and Editing

Command History

Viewing the Command History

Environment Variables for History

History Expansion

Basic History Expansion

Referencing Specific Commands

Using Substring Matches

Modifying Previous Commands

Using !$ and !*

Safety with :p

Command-Line Editing

Cursoring Within a Command

Incremental Search

4. Navigating the Filesystem with Speed

Efficient Directory Navigation

Using cd

CDPATH

Organizing Your Home Directory

Strategies for Fast Navigation

Using pushd and popd for Directory Management

Basic Usage

Example Workflow

Conclusion

Consolidating Multiple SFTP Accounts Into One Master Account

SFTP account

Mount multiple accounts to one account

Files at one location

Summary

Making sense of XML/JSON items in the shell

XML

JSON

Shell Command Outputs Truncated in Python

Problem

Root Cause

Solution

Explanation

Conclusion

How to check process duration in Linux with the “ps” command

Age comparison in Bash for files and processes

Shell efficiency: mkdir and mv

Using Awk to beautify grep searches

Search in files tools

Understanding the Shell with `ls`

Using `!$` and `!*`

Safety with `:p`

Using `cd`