https://www.endpointdev.com/blog/tags/tools/2023-07-05T00:00:00+00:00End Point DevUnix tools & tipshttps://www.endpointdev.com/blog/2023/07/unix-tools/2023-07-05T00:00:00+00:00Muhammad Najmi bin Ahmad Zabidi
<p><img src="/blog/2023/07/unix-tools/idaho-mountains.webp" alt="The view from a mountain ridge. the sky is light blue and partially covered in clouds. Green ridges covered in pine trees lead down to a flat valley populated by a small town."></p>
<!-- Photo by Seth Jensen, 2023. -->
<p>The command line interface (CLI) programs/tools found on UNIX-like (*NIX) systems are among the most useful tools available to system administrators. CLI tools allow a system administrator to handle the *NIX systems either remotely or locally without needing to install Graphical User Interface (GUI) packages.</p>
<p>Almost twenty years ago, I bought two O’Reilly books: “sed & awk, second edition,” written by Dale Dougherty & Arnold Robbins, and “Mastering Regular Expressions,” written by Jeffrey E.F. Friedl. These two books were printed in the late ’90s, and I bought them in 2004. I remembered them recently when my co-workers and I held a few study sessions to learn regular expressions. When a page and chapter are mentioned in this post, I’m referring to the editions of these books printed in that particular year.</p>
<p>In this blog post, I’ll detail some common use cases I encounter day to day, as well as some related tools that help me handle them.</p>
<h3 id="sed">sed</h3>
<p>During my regular work as a system administrator, I usually use <strong>sed</strong> (“<strong>s</strong>tream <strong>ed</strong>itor”) to do string replacement across files and <strong>awk</strong> for log files or file analysis with arbitrary strings as the input field separator using the -F flag.</p>
<p>One common use case for sed I’ve found is updating config files for the Icinga monitoring system. Let’s say that we have a file named server101.cfg and we want to use the same config for server 102. One way to solve this on the command line is doing a simple search and replace with sed:</p>
<pre tabindex="0"><code class="language-console" data-lang="console">$ cp server101.cfg server102.cfg
$ sed 's/server101/server102/g' server102.cfg
$ icinga -v /etc/icinga/icinga.cfg # This will check that Icinga is able to understand the newly copied file
</code></pre><p>Make sure you use diff or another similar tool to make sure this doesn’t accidentally modify anything it’s not supposed to. You can get more detailed with your regular expressions to avoid such issues, but that’s beyond the scope of this post.</p>
<h3 id="awk">awk</h3>
<p>Sometimes I need to use awk for scripting, but most of the time, I use it for parsing things like IP addresses in log files, or any other string in logs. For example, you can check for plaintext passwords or credit card details in places they’re not supposed to be.</p>
<p>The sed & awk book started with the introduction of ed, an ancient line editor. Then it touched on the field separator, which is represented by the <code>-F</code> flag.</p>
<p>In the following example, I ran the <code>blkid</code> command in order to check the UUID of the SCSI devices that connected to my computer, then used awk to only show the output I needed.</p>
<pre tabindex="0"><code class="language-console" data-lang="console">$ sudo blkid /dev/sd{d,e,g}
/dev/sdd: UUID="7eb6302f-e727-4433-8c49-8a7842d18e1e" TYPE="crypto_LUKS"
/dev/sde: UUID="68b2382e-13b8-4bdb-a6cb-15f6844d464b" TYPE="crypto_LUKS"
/dev/sdg: UUID="7237cc7d-0483-4c2a-a503-a11ea88b3690" TYPE="crypto_LUKS"
</code></pre><pre tabindex="0"><code class="language-console" data-lang="console">$ sudo blkid /dev/sd{d,e,g} | awk -F "\"" {'print $2'}
7eb6302f-e727-4433-8c49-8a7842d18e1e
68b2382e-13b8-4bdb-a6cb-15f6844d464b
7237cc7d-0483-4c2a-a503-a11ea88b3690
</code></pre><p>During my day-to-day work I seldom use both <code>sed</code> and <code>awk</code> together, but there are many situations where it can be useful to do so. See page 23 of the sed & awk book for one such example where both of these tools can be used together.</p>
<h3 id="using-regular-expressions-in-nix-tools">Using regular expressions in *NIX tools</h3>
<p>The sed & awk book touches on regular expression syntax, since system administrators might have to use regular expressions in cases where large/repetitive tasks are involved. I do not remember who suggested buying this pair of books, but I am really grateful for the person’s suggestion.</p>
<p>To search a file using regular expressions, I use the program “grep.” I often use the <code>-r</code> flag for recursive parsing, the <code>-w</code> flag to match whole words only, and the <code>--color=always</code> flag to colorize the terminal output. Recent versions of GNU grep use the <code>--color=auto</code> option by default so I don’t have to explicitly specify it when sending output straight to a terminal. We can also use <code>--color=never</code> if we want to remove the color matching.</p>
<pre tabindex="0"><code class="language-console" data-lang="console">$ grep 1 -w --color=never tmp/file.txt
Mon Aug 1 11:30:01 AM +08 2022
Thu Sep 1 11:30:01 AM +08 2022
Sat Oct 1 11:30:01 AM +08 2022
Tue Nov 1 11:30:01 AM +08 2022
Thu Dec 1 11:30:01 AM +08 2022
Sun Jan 1 11:30:01 AM +08 2023
Wed Feb 1 11:30:01 AM +08 2023
Wed Mar 1 11:30:01 AM +08 2023
</code></pre><p>To extend the capability of grep for regular expression’s usage, I used <code>-E</code>, but then I learned about the <code>-P</code> flag, which lets you use Perl-compatible regular expressions (PCREs), which I prefer.</p>
<p>Say I have a file named <code>fruit.txt</code> with the following contents:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">apple banana orange
apple orange papaya
durian rambutan
starfruit
</code></pre></div><p>To find lines starting with “durian”:</p>
<pre tabindex="0"><code class="language-console" data-lang="console">$ grep -P '^durian' fruit.txt
durian rambutan
</code></pre><p>To find lines ending with “papaya”:</p>
<pre tabindex="0"><code class="language-console" data-lang="console">$ grep -P "papaya$" fruit.txt
apple orange papaya
</code></pre><p>To find lines with three words separated by two spaces:</p>
<pre tabindex="0"><code class="language-console" data-lang="console">$ grep -P '\w+ \w+ \w+' fruit.txt
apple banana orange
apple orange papaya
</code></pre><p>To find all lines containing a word starting with “a” or “d”:</p>
<pre tabindex="0"><code class="language-console" data-lang="console">$ grep -P '\b(a|d)\w+' fruit.txt
apple banana orange
apple orange papaya
durian rambutan
</code></pre><p>These are just a few examples of what <code>grep</code> can do; it’s very capable especially when you use PCREs.</p>
<p>Apart from sed, awk and grep, there are many other useful tools for text processing. Some that I commonly use are the sort and uniq programs.</p>
<h3 id="sort">sort</h3>
<p>sort and uniq are frequently used together. sort, as its name implies, is used to sort the parsed text per whatever options are provided.</p>
<p>Assume that we have the following file, “prices.csv”:</p>
<pre tabindex="0"><code class="language-console" data-lang="console">$ cat prices.csv
chicken,$1.50
duck,$1.20
beef,$6.10
chicken,$1.50
lamb,$3.20
fish,$4.09
</code></pre><p>Using the <code>sort</code> command, we could sort the items according a specified text separator (<code>-t</code>) and the column (key) that we want to prioritize (<code>-k</code>). Here, I want to sort the item based on data in the second column:</p>
<pre tabindex="0"><code class="language-console" data-lang="console">$ sort -t "," -k 2 prices.csv
duck,$1.20
chicken,$1.50
chicken,$1.50
lamb,$3.20
fish,$4.09
beef,$6.10
</code></pre><p>Without the other parameters, sort will just sort the lines alphabetically.</p>
<pre tabindex="0"><code class="language-console" data-lang="console">$ sort prices.csv
beef,$6.10
chicken,$1.50
chicken,$1.50
duck,$1.20
fish,$4.09
lamb,$3.20
</code></pre><p>We can use uniq to eliminate repeated lines. Running <code>uniq</code> without sorting will have no effect, since the duplicate lines <code>chicken,$1.50</code> do not appear sequentially:</p>
<pre tabindex="0"><code class="language-console" data-lang="console">$ uniq prices.csv
chicken,$1.50
duck,$1.20
beef,$6.10
chicken,$1.50
lamb,$3.20
fish,$4.09
</code></pre><p>To merge them and clean up the data, we can sort first:</p>
<pre tabindex="0"><code class="language-console" data-lang="console">$ sort prices.csv | uniq
beef,$6.10
chicken,$1.50
duck,$1.20
fish,$4.09
lamb,$3.20
</code></pre><h3 id="find">find</h3>
<p>Say I want to search a full hard disk to check for files that are not needed anymore. I would use the following command, with each option detailed below the output:</p>
<pre tabindex="0"><code class="language-console" data-lang="console">$ find . -maxdepth 1 -mtime +3000 -type d -exec ls {} -ld \;
drwxrwxrwx 2 najmi najmi 4096 Nov 25 2012 ./Terminal
drwxrwxrwx 3 najmi najmi 4096 Dis 27 2012 ./gnome-disk-utility
drwxrwxrwx 8 najmi najmi 4096 Nov 27 2012 ./xfce4
drwxrwxrwx 2 najmi najmi 4096 Nov 25 2012 ./libaccounts-glib
drwxrwxrwx 2 najmi najmi 4096 Dis 1 2012 ./sakura
drwxrwxrwx 2 najmi najmi 4096 Nov 25 2012 ./software-center
drwxrwxrwx 4 najmi najmi 4096 Nov 25 2012 ./evolution
drwxrwxrwx 2 najmi najmi 4096 Nov 25 2012 ./update-notifier
drwxrwxrwx 2 najmi najmi 4096 Dis 22 2012 ./Thunar
drwxrwxrwx 3 najmi najmi 4096 Nov 25 2012 ./compiz-1
drwxrwxrwx 2 najmi najmi 4096 Dis 28 2012 ./tracker
drwxrwxrwx 3 najmi najmi 4096 Dis 9 2012 ./menus
drwxrwxrwx 3 najmi najmi 4096 Dis 27 2012 ./gnome-control-center
drwxrwxrwx 2 najmi najmi 4096 Nov 25 2012 ./goa-1.0
drwxrwxrwx 2 najmi najmi 4096 Nov 28 2012 ./enchant
drwxrwxrwx 2 najmi najmi 4096 Dis 8 2012 ./Pinta
drwxrwxrwx 2 najmi najmi 4096 Nov 25 2012 ./gmusicbrowser
drwxrwxrwx 3 najmi najmi 4096 Dis 14 2012 ./audacious
drwxrwxrwx 3 najmi najmi 4096 Dis 27 2012 ./gnome-session
drwxrwxrwx 5 najmi najmi 4096 Nov 28 2012 ./mate
drwxrwxrwx 2 najmi najmi 4096 Nov 25 2012 ./ristretto
drwxrwxrwx 31 najmi najmi 4096 Dis 24 2012 ./chromium
drwxrwxrwx 5 najmi najmi 4096 Dis 8 2012 ./mono.addins
drwxrwxrwx 3 najmi najmi 4096 Nov 25 2012 ./mate-session
drwxrwxrwx 3 najmi najmi 4096 Nov 25 2012 ./ibus
drwxrwxrwx 4 najmi najmi 4096 Nov 25 2012 ./libreoffice
drwxrwxrwx 3 najmi najmi 4096 Nov 25 2012 ./caja
drwxrwxrwx 2 najmi najmi 4096 Dis 18 2012 ./Empathy
</code></pre><ul>
<li>In this case I search in the current directory with the dot <code>.</code> notation.</li>
<li>The <code>-maxdepth 1</code> option means the search task must not be done beyond the current directory.</li>
<li><code>-mtime +3000</code> means only show directories that have not been modified within the last 3000 days.</li>
<li><code>-type d</code> means match directories, but not files (to only match files, use the <code>-type f</code> flag).</li>
<li>The <code>-exec ls {} -ld \;</code> arguments mean it will run the <code>ls -ld</code> command for each match. This will show the modification time of the directory passed by the <code>{}</code> placeholder, which is replaced by the file name of the current search result.</li>
</ul>
<blockquote>
<p>Note: In Linux with GNU tools it is possible to order the flags like either <code>ls {} -ld</code> or <code>ls -ld {}</code>, but I realized that in some other *NIX variants, you always need to put the flag before the variable/parameter; that is, <code>ls -ld {}</code>.</p>
</blockquote>
<h3 id="conclusion">Conclusion</h3>
<p>There are newer tools which may be provided by more recent Linux distros or BSD variants, but in this short write-up I wanted to stick to the basics which are useful to anyone using *NIX. It is more than worth your time to learn these tools in detail.</p>
<blockquote>
<p>For further reading, I would also recommend “UNIX and Linux System Administration Handbook” by Nemeth, E., Snyder, G., Hein, T., Whaley, B., & Mackin, D. (2020).</p>
</blockquote>
Aligning monospace font text columns with an old Unix toolhttps://www.endpointdev.com/blog/2022/05/aligning-fixed-width-font-text-columns/2022-05-30T00:00:00+00:00Jon Jensen
<p><img src="/blog/2022/05/aligning-fixed-width-font-text-columns/20220316_183928.webp" alt="Photo of old wooden bridge with moss on it">
Photo by Garrett Skinner</p>
<h3 id="a-blast-from-1990-column">A blast from 1990: column</h3>
<p>A while back I learned of a nice old Unix command-line tool called <code>column</code>. It first appeared in 4.3BSD-Reno, released in July 1990. (This is not to be confused with the different, even older Unix tool <code>col</code>.)</p>
<p><code>column</code> formats plain text into nice columns based on the width of the input separated by tabs or groups of spaces.</p>
<p>For example, take this mess found in a server’s <code>/etc/fstab</code> file defining filesystem mount points. It is a real example lightly redacted to remove business details. You may need to scroll right to see the end of the fairly long lines:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">/data3/customer_uploads /home/interch/htdocs/shared/customer_uploads none rw,bind 0 0
/data3/customer_images /home/interch/htdocs/shared/customer_images none rw,bind 0 0
/data3/images/items home/interch/htdocs/images/items none rw,bind 0 0
/data3/images/thumb home/interch/htdocs/images/thumb none rw,bind 0 0
/data3/upload_images /home/interch/upload_images none rw,bind 0 0
/data3/design /home/interch/htdocs/shared/design none rw,bind 0 0
/data3/design_temp /home/interch/htdocs/shared/design_temp none rw,bind 0 0
/data3/reports/cat1 /home/interch/htdocs/cat-numero-uno/images/reports none rw,bind 0 0
/data3/reports/cat2 /home/interch/htdocs/cat-zahl-zwei/images/reports none rw,bind 0 0
/data3/reports/cat3 /home/interch/htdocs/cat-number-three/images/reports none rw,bind 0 0
/data3/reports/cat4 /home/interch/htdocs/cat-quatre/images/reports none rw,bind 0 0
/data3/reports/cat5 /home/interch/htdocs/cat-pět/images/reports none rw,bind 0 0
/data3/shared_var /home/interch/catalogs/shared/var none rw,bind 0 0
</code></pre></div><p>That is really unsightly and includes a mix of spaces and tabs.</p>
<p>If we feed it to <code>column -t</code> we get the same data aligned much nicer:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">% column -t /etc/fstab
/data3/customer_uploads /home/interch/htdocs/shared/customer_uploads none rw,bind 0 0
/data3/customer_images /home/interch/htdocs/shared/customer_images none rw,bind 0 0
/data3/images/items home/interch/htdocs/images/items none rw,bind 0 0
/data3/images/thumb home/interch/htdocs/images/thumb none rw,bind 0 0
/data3/upload_images /home/interch/upload_images none rw,bind 0 0
/data3/design /home/interch/htdocs/shared/design none rw,bind 0 0
/data3/design_temp /home/interch/htdocs/shared/design_temp none rw,bind 0 0
/data3/reports/cat1 /home/interch/htdocs/cat-numero-uno/images/reports none rw,bind 0 0
/data3/reports/cat2 /home/interch/htdocs/cat-zahl-zwei/images/reports none rw,bind 0 0
/data3/reports/cat3 /home/interch/htdocs/cat-number-three/images/reports none rw,bind 0 0
/data3/reports/cat4 /home/interch/htdocs/cat-quatre/images/reports none rw,bind 0 0
/data3/reports/cat5 /home/interch/htdocs/cat-pět/images/reports none rw,bind 0 0
/data3/shared_var /home/interch/catalogs/shared/var none rw,bind 0 0
</code></pre></div><p>That isn’t just prettier! It also makes some things stand out prominently at a glance:</p>
<ul>
<li>In the second column there are two mount points not starting with <code>/</code>.</li>
<li>It’s easy to see that most of the paths in the second column start with <code>/home/interch/htdocs/</code> and the few that don’t, stand out.</li>
<li>The final 4 columns are all identical, which was unclear in the unaligned original.</li>
</ul>
<h3 id="text-tables-to-json">Text tables to JSON</h3>
<p>The <a href="https://en.wikipedia.org/wiki/Util-linux">util-linux</a> version of <code>column</code> includes extra options beyond the original. One very useful one is <code>-J</code> or <code>--json</code> which produces a JSON object for each row based on column names you define in the argument <code>-N</code> or <code>--table-columns</code>.</p>
<p>Reviewing <a href="https://www.man7.org/linux/man-pages/man5/fstab.5.html"><code>man fstab</code></a> for details on what each column is used for in the sample above, we can instruct <code>column</code> to produce JSON output like this:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-sh" data-lang="sh">% column -J -n fstab -N spec,file,vfstype,mntops,freq,passno /etc/fstab
</code></pre></div><p>Which gives us this result:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-json" data-lang="json">{
<span style="color:#b06;font-weight:bold">"fstab"</span>: [
{
<span style="color:#b06;font-weight:bold">"spec"</span>: <span style="color:#d20;background-color:#fff0f0">"/data3/customer_uploads"</span>,
<span style="color:#b06;font-weight:bold">"file"</span>: <span style="color:#d20;background-color:#fff0f0">"/home/interch/htdocs/shared/customer_uploads"</span>,
<span style="color:#b06;font-weight:bold">"vfstype"</span>: <span style="color:#d20;background-color:#fff0f0">"none"</span>,
<span style="color:#b06;font-weight:bold">"mntops"</span>: <span style="color:#d20;background-color:#fff0f0">"rw,bind"</span>,
<span style="color:#b06;font-weight:bold">"freq"</span>: <span style="color:#d20;background-color:#fff0f0">"0"</span>,
<span style="color:#b06;font-weight:bold">"passno"</span>: <span style="color:#d20;background-color:#fff0f0">"0"</span>
},{
<span style="color:#b06;font-weight:bold">"spec"</span>: <span style="color:#d20;background-color:#fff0f0">"/data3/customer_images"</span>,
<span style="color:#b06;font-weight:bold">"file"</span>: <span style="color:#d20;background-color:#fff0f0">"/home/interch/htdocs/shared/customer_images"</span>,
<span style="color:#b06;font-weight:bold">"vfstype"</span>: <span style="color:#d20;background-color:#fff0f0">"none"</span>,
<span style="color:#b06;font-weight:bold">"mntops"</span>: <span style="color:#d20;background-color:#fff0f0">"rw,bind"</span>,
<span style="color:#b06;font-weight:bold">"freq"</span>: <span style="color:#d20;background-color:#fff0f0">"0"</span>,
<span style="color:#b06;font-weight:bold">"passno"</span>: <span style="color:#d20;background-color:#fff0f0">"0"</span>
},{
<span style="color:#b06;font-weight:bold">"spec"</span>: <span style="color:#d20;background-color:#fff0f0">"/data3/images/items"</span>,
<span style="color:#b06;font-weight:bold">"file"</span>: <span style="color:#d20;background-color:#fff0f0">"home/interch/htdocs/images/items"</span>,
<span style="color:#b06;font-weight:bold">"vfstype"</span>: <span style="color:#d20;background-color:#fff0f0">"none"</span>,
<span style="color:#b06;font-weight:bold">"mntops"</span>: <span style="color:#d20;background-color:#fff0f0">"rw,bind"</span>,
<span style="color:#b06;font-weight:bold">"freq"</span>: <span style="color:#d20;background-color:#fff0f0">"0"</span>,
<span style="color:#b06;font-weight:bold">"passno"</span>: <span style="color:#d20;background-color:#fff0f0">"0"</span>
},{
<span style="color:#b06;font-weight:bold">"spec"</span>: <span style="color:#d20;background-color:#fff0f0">"/data3/images/thumb"</span>,
<span style="color:#b06;font-weight:bold">"file"</span>: <span style="color:#d20;background-color:#fff0f0">"home/interch/htdocs/images/thumb"</span>,
<span style="color:#b06;font-weight:bold">"vfstype"</span>: <span style="color:#d20;background-color:#fff0f0">"none"</span>,
<span style="color:#b06;font-weight:bold">"mntops"</span>: <span style="color:#d20;background-color:#fff0f0">"rw,bind"</span>,
<span style="color:#b06;font-weight:bold">"freq"</span>: <span style="color:#d20;background-color:#fff0f0">"0"</span>,
<span style="color:#b06;font-weight:bold">"passno"</span>: <span style="color:#d20;background-color:#fff0f0">"0"</span>
},{
<span style="color:#b06;font-weight:bold">"spec"</span>: <span style="color:#d20;background-color:#fff0f0">"/data3/upload_images"</span>,
<span style="color:#b06;font-weight:bold">"file"</span>: <span style="color:#d20;background-color:#fff0f0">"/home/interch/upload_images"</span>,
<span style="color:#b06;font-weight:bold">"vfstype"</span>: <span style="color:#d20;background-color:#fff0f0">"none"</span>,
<span style="color:#b06;font-weight:bold">"mntops"</span>: <span style="color:#d20;background-color:#fff0f0">"rw,bind"</span>,
<span style="color:#b06;font-weight:bold">"freq"</span>: <span style="color:#d20;background-color:#fff0f0">"0"</span>,
<span style="color:#b06;font-weight:bold">"passno"</span>: <span style="color:#d20;background-color:#fff0f0">"0"</span>
},{
<span style="color:#b06;font-weight:bold">"spec"</span>: <span style="color:#d20;background-color:#fff0f0">"/data3/design"</span>,
<span style="color:#b06;font-weight:bold">"file"</span>: <span style="color:#d20;background-color:#fff0f0">"/home/interch/htdocs/shared/design"</span>,
<span style="color:#b06;font-weight:bold">"vfstype"</span>: <span style="color:#d20;background-color:#fff0f0">"none"</span>,
<span style="color:#b06;font-weight:bold">"mntops"</span>: <span style="color:#d20;background-color:#fff0f0">"rw,bind"</span>,
<span style="color:#b06;font-weight:bold">"freq"</span>: <span style="color:#d20;background-color:#fff0f0">"0"</span>,
<span style="color:#b06;font-weight:bold">"passno"</span>: <span style="color:#d20;background-color:#fff0f0">"0"</span>
},{
<span style="color:#b06;font-weight:bold">"spec"</span>: <span style="color:#d20;background-color:#fff0f0">"/data3/design_temp"</span>,
<span style="color:#b06;font-weight:bold">"file"</span>: <span style="color:#d20;background-color:#fff0f0">"/home/interch/htdocs/shared/design_temp"</span>,
<span style="color:#b06;font-weight:bold">"vfstype"</span>: <span style="color:#d20;background-color:#fff0f0">"none"</span>,
<span style="color:#b06;font-weight:bold">"mntops"</span>: <span style="color:#d20;background-color:#fff0f0">"rw,bind"</span>,
<span style="color:#b06;font-weight:bold">"freq"</span>: <span style="color:#d20;background-color:#fff0f0">"0"</span>,
<span style="color:#b06;font-weight:bold">"passno"</span>: <span style="color:#d20;background-color:#fff0f0">"0"</span>
},{
<span style="color:#b06;font-weight:bold">"spec"</span>: <span style="color:#d20;background-color:#fff0f0">"/data3/reports/cat1"</span>,
<span style="color:#b06;font-weight:bold">"file"</span>: <span style="color:#d20;background-color:#fff0f0">"/home/interch/htdocs/cat-numero-uno/images/reports"</span>,
<span style="color:#b06;font-weight:bold">"vfstype"</span>: <span style="color:#d20;background-color:#fff0f0">"none"</span>,
<span style="color:#b06;font-weight:bold">"mntops"</span>: <span style="color:#d20;background-color:#fff0f0">"rw,bind"</span>,
<span style="color:#b06;font-weight:bold">"freq"</span>: <span style="color:#d20;background-color:#fff0f0">"0"</span>,
<span style="color:#b06;font-weight:bold">"passno"</span>: <span style="color:#d20;background-color:#fff0f0">"0"</span>
},{
<span style="color:#b06;font-weight:bold">"spec"</span>: <span style="color:#d20;background-color:#fff0f0">"/data3/reports/cat2"</span>,
<span style="color:#b06;font-weight:bold">"file"</span>: <span style="color:#d20;background-color:#fff0f0">"/home/interch/htdocs/cat-zahl-zwei/images/reports"</span>,
<span style="color:#b06;font-weight:bold">"vfstype"</span>: <span style="color:#d20;background-color:#fff0f0">"none"</span>,
<span style="color:#b06;font-weight:bold">"mntops"</span>: <span style="color:#d20;background-color:#fff0f0">"rw,bind"</span>,
<span style="color:#b06;font-weight:bold">"freq"</span>: <span style="color:#d20;background-color:#fff0f0">"0"</span>,
<span style="color:#b06;font-weight:bold">"passno"</span>: <span style="color:#d20;background-color:#fff0f0">"0"</span>
},{
<span style="color:#b06;font-weight:bold">"spec"</span>: <span style="color:#d20;background-color:#fff0f0">"/data3/reports/cat3"</span>,
<span style="color:#b06;font-weight:bold">"file"</span>: <span style="color:#d20;background-color:#fff0f0">"/home/interch/htdocs/cat-number-three/images/reports"</span>,
<span style="color:#b06;font-weight:bold">"vfstype"</span>: <span style="color:#d20;background-color:#fff0f0">"none"</span>,
<span style="color:#b06;font-weight:bold">"mntops"</span>: <span style="color:#d20;background-color:#fff0f0">"rw,bind"</span>,
<span style="color:#b06;font-weight:bold">"freq"</span>: <span style="color:#d20;background-color:#fff0f0">"0"</span>,
<span style="color:#b06;font-weight:bold">"passno"</span>: <span style="color:#d20;background-color:#fff0f0">"0"</span>
},{
<span style="color:#b06;font-weight:bold">"spec"</span>: <span style="color:#d20;background-color:#fff0f0">"/data3/reports/cat4"</span>,
<span style="color:#b06;font-weight:bold">"file"</span>: <span style="color:#d20;background-color:#fff0f0">"/home/interch/htdocs/cat-quatre/images/reports"</span>,
<span style="color:#b06;font-weight:bold">"vfstype"</span>: <span style="color:#d20;background-color:#fff0f0">"none"</span>,
<span style="color:#b06;font-weight:bold">"mntops"</span>: <span style="color:#d20;background-color:#fff0f0">"rw,bind"</span>,
<span style="color:#b06;font-weight:bold">"freq"</span>: <span style="color:#d20;background-color:#fff0f0">"0"</span>,
<span style="color:#b06;font-weight:bold">"passno"</span>: <span style="color:#d20;background-color:#fff0f0">"0"</span>
},{
<span style="color:#b06;font-weight:bold">"spec"</span>: <span style="color:#d20;background-color:#fff0f0">"/data3/reports/cat5"</span>,
<span style="color:#b06;font-weight:bold">"file"</span>: <span style="color:#d20;background-color:#fff0f0">"/home/interch/htdocs/cat-pět/images/reports"</span>,
<span style="color:#b06;font-weight:bold">"vfstype"</span>: <span style="color:#d20;background-color:#fff0f0">"none"</span>,
<span style="color:#b06;font-weight:bold">"mntops"</span>: <span style="color:#d20;background-color:#fff0f0">"rw,bind"</span>,
<span style="color:#b06;font-weight:bold">"freq"</span>: <span style="color:#d20;background-color:#fff0f0">"0"</span>,
<span style="color:#b06;font-weight:bold">"passno"</span>: <span style="color:#d20;background-color:#fff0f0">"0"</span>
},{
<span style="color:#b06;font-weight:bold">"spec"</span>: <span style="color:#d20;background-color:#fff0f0">"/data3/shared_var"</span>,
<span style="color:#b06;font-weight:bold">"file"</span>: <span style="color:#d20;background-color:#fff0f0">"/home/interch/catalogs/shared/var"</span>,
<span style="color:#b06;font-weight:bold">"vfstype"</span>: <span style="color:#d20;background-color:#fff0f0">"none"</span>,
<span style="color:#b06;font-weight:bold">"mntops"</span>: <span style="color:#d20;background-color:#fff0f0">"rw,bind"</span>,
<span style="color:#b06;font-weight:bold">"freq"</span>: <span style="color:#d20;background-color:#fff0f0">"0"</span>,
<span style="color:#b06;font-weight:bold">"passno"</span>: <span style="color:#d20;background-color:#fff0f0">"0"</span>
}
]
}
</code></pre></div><p>JSON takes a lot more room, and of course is not the format Linux expects for this particular file, but transforming tabular data to JSON format in other situations can be more readable for exchange across different systems since each field is labeled, and unused fields can be omitted. Plus JSON syntax is rigorously defined, nested data structures are possible, etc.</p>
<h3 id="columnizing-lists">Columnizing lists</h3>
<p>All versions of <code>column</code> can also columnize lists, either horizontally (across) or vertically (down). Take this list of people’s names:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">Amy
Anna
Bob
Brenda
Cameron
Doug
Emily
Frank
Jane
Jill
Jim
Joe
John
Karen
Kate
Liz
Mary
Mike
Sarah
Steve
Victoria
</code></pre></div><p>Put that in file <code>/tmp/names</code> and <code>column</code> will format it in columns fitting the width of your terminal:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">% column /tmp/names
Amy Cameron Jane John Mary Victoria
Anna Doug Jill Karen Mike
Bob Emily Jim Kate Sarah
Brenda Frank Joe Liz Steve
</code></pre></div><p>It uses 2 or more tab characters to separate the columns, based on standard terminal 8-space tab stops, so the above doesn’t look right here on the web.</p>
<p>What appears in my terminal looks like:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">% column /tmp/names
Amy Cameron Jane John Mary Victoria
Anna Doug Jill Karen Mike
Bob Emily Jim Kate Sarah
Brenda Frank Joe Liz Steve
</code></pre></div><p>Or you can use <code>column -t</code> that we discussed earlier to format the columns more compactly with spaces:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">% column /tmp/names | column -t
Amy Cameron Jane John Mary Victoria
Anna Doug Jill Karen Mike
Bob Emily Jim Kate Sarah
Brenda Frank Joe Liz Steve
</code></pre></div><p>You can also ask for the list to be delivered horizontally, rather than vertically, with <code>column -x</code>:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">% column -x /tmp/names | column -t
Amy Anna Bob Brenda Cameron Doug
Emily Frank Jane Jill Jim Joe
John Karen Kate Liz Mary Mike
Sarah Steve Victoria
</code></pre></div><p>Note that it isn’t doing anything to affect your ordering. You can order the lines in your original file however you want and it will preserve them. But other tools can help you here: Use <code>sort -u</code> to sort alphabetically and remove duplicates.</p>
<p>For more options and details see the <code>column</code> man page for the <a href="https://man7.org/linux/man-pages/man1/column.1.html">util-linux column</a> version or <a href="https://www.freebsd.org/cgi/man.cgi?query=column&apropos=0&sektion=0&manpath=FreeBSD+13.1-RELEASE+and+Ports&arch=default&format=html">FreeBSD column</a> version (same as macOS).</p>
<h3 id="a-blast-from-1974-pr">A blast from 1974: pr</h3>
<p>There is an even older Unix tool for columnizing lists in the same way. It is called <code>pr</code> and dates to First Edition Unix in 1971, but did not gain the options we are using here until Fifth Edition Unix in 1974 as seen in the <a href="https://minnie.tuhs.org/cgi-bin/utree.pl?file=V6/usr/man/man1/pr.1">V6 pr man page</a>.</p>
<p>We need to tell it how many columns to produce, so we will ask for 6 columns as <code>column</code> was doing above. Note that <code>pr</code> emits a curious mix of tabs and spaces, which <code>cat -t</code> reveals here as <code>^I</code> (since a tab is the same thing as Control+I):</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">% pr -t -6 /tmp/names | cat -t
Amy^I Cameron^IJane^I John^ILiz^I Sarah
Anna^I Doug^IJill^I Karen^IMary^I Steve
Bob^I Emily^IJim^I Kate^IMike^I Victoria
Brenda^I Frank^IJoe
</code></pre></div><p>But in a terminal it looks fine:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">% pr -t -6 /tmp/names
Amy Cameron Jane John Liz Sarah
Anna Doug Jill Karen Mary Steve
Bob Emily Jim Kate Mike Victoria
Brenda Frank Joe
</code></pre></div><p>See the many more options of pr in the <a href="https://man7.org/linux/man-pages/man1/pr.1.html">GNU coreutils pr man page</a> and <a href="https://www.freebsd.org/cgi/man.cgi?query=pr&apropos=0&sektion=0&manpath=FreeBSD+13.1-RELEASE+and+Ports&arch=default&format=html">FreeBSD pr man page</a> (same as macOS).</p>
<h3 id="a-blast-from-1979-expand">A blast from 1979: expand</h3>
<p>A useful tool for dealing with tabs is <code>expand</code>, which <a href="https://github.com/dank101/3BSD/blob/master/cmd/expand.c">first appeared in 3BSD</a> in 1979. (Despite the FreeBSD and macOS man pages saying it appeared in 1BSD, I don’t see it there or in 2BSD.)</p>
<p>We can use it to convert tabs to spaces just like a terminal would:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">% pr -t -6 /tmp/names | expand | cat -t
Amy Cameron Jane John Liz Sarah
Anna Doug Jill Karen Mary Steve
Bob Emily Jim Kate Mike Victoria
Brenda Frank Joe
</code></pre></div><p><code>expand</code> accepts the optional <code>-t</code> argument with a number to use as tabstop width instead of the default 8.</p>
<h3 id="and-unexpand">And unexpand</h3>
<p>I have used <code>expand</code> for years, but only recently learned of <code>unexpand</code> which goes the other way, converting runs of spaces into tabs:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">% pr -t -6 /tmp/names | expand | unexpand -a
Amy Cameron Jane John Liz Sarah
Anna Doug Jill Karen Mary Steve
Bob Emily Jim Kate Mike Victoria
Brenda Frank Joe
</code></pre></div><p>It looks fine in the terminal, but let’s see if it actually used tabs:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">% pr -t -6 /tmp/names | expand | unexpand -a | cat -t
Amy^I Cameron^IJane^I John^ILiz^I Sarah
Anna^I Doug^IJill^I Karen^IMary^I Steve
Bob^I Emily^IJim^I Kate^IMike^I Victoria
Brenda^I Frank^IJoe
</code></pre></div><p>Interesting… For some reason <code>unexpand</code> doesn’t actually convert all the spaces to tabs, but just one initial tab per gutter between columns. Running the output through <code>unexpand -a</code> again has no further effect. Strange.</p>
<p>The GNU coreutils version of <code>unexpand</code> that lives in most Linux systems is what I would call bug-compatible with the BSD version in this regard.</p>
<p>Oh, well. There’s probably a reason.</p>
<p>Since it has handled turning the initial tabstop’s varying number of spaces into a tab, we can easily remove the remaining fixed multiples of spaces on our own for a more compact list:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">% pr -t -6 /tmp/names | expand | unexpand -a | sed 's/ //g'
Amy Cameron Jane John Liz Sarah
Anna Doug Jill Karen Mary Steve
Bob Emily Jim Kate Mike Victoria
Brenda Frank Joe
</code></pre></div><h3 id="ready-at-hand">Ready at hand</h3>
<p><code>column</code>, <code>pr</code>, <code>expand</code>, and <code>unexpand</code> all come with most Linux and BSD systems, including macOS. It is amazing what great old tools are on many of our computers all the time, waiting to be used!</p>
<p>You can use these programs as filters inside your favorite text editor or IDE. For example, to achieve the same list columnization as above you can select a block of text and send it to external commands:</p>
<ul>
<li>In Vim, visually select text with <code>v</code> or <code>V</code> and then type <code>!column | expand</code> and press Enter.</li>
<li>In VS Code you can install an extension called Filter Text (by yhirose). Once installed, type Control+K Control+F (⌘K ⌘F on macOS) and then <code>column | expand</code> and Enter.</li>
<li>In IntelliJ IDEA you can install a plugin called Shell Filter (by Dennis Plöger). Once installed, select your text, then choose menu item Edit > Custom Shell Filter and then type <code>column -c 80 | expand</code> and Enter.</li>
<li>Most other editors have a way to do this too. Search for “filter”, “pipe”, and/or “command”.</li>
</ul>
<p>Your selection will be replaced by the output.</p>
<h3 id="unicode">Unicode</h3>
<p>These tools come from an era when one byte of input resulted in one visual character, so for any UTF-8 characters outside the limited classic ASCII character set, we could expect old tools may miscalculate the needed space between columns.</p>
<p>Originally in my testing they appeared not to be aware of some Unicode characters’ width, perhaps due to combining characters or alternate forms. But surprisingly they have been modernized and seem to work fine with most Unicode characters such as Latin letters with diacritics, letters in other alphabets, Chinese characters, and even emoji! 😊</p>
Formatting SQL code with pgFormatter within Vimhttps://www.endpointdev.com/blog/2022/04/formatting-sql-vim-pgformat/2022-04-26T00:00:00+00:00Josh Tolley
<p><img src="/blog/2022/04/formatting-sql-vim-pgformat/20220411_091408.webp" alt="Outdoor view of a creek bank with dry trees and old wooden buildings against a blue sky">
Photo by Garrett Skinner</p>
<p>Sometimes a little, seemingly simple tip can make a world of difference. I’ve got enough gray hair these days that it would be pretty easy for me to start thinking I’d seen an awful lot, yet quite frequently when I watch a colleague working in a meeting or a <a href="https://github.com/tmux/tmux">tmux</a> session or somewhere, I learn some new and simple thing that makes my life demonstrably easier.</p>
<p><a href="https://fluca1978.github.io/2022/04/13/EmacsPgFormatter.html">Luca Ferrari recently authored a post</a> about using pgFormatter in Emacs; essentially the same thing works in Vim, my editor of choice, and it’s one of my favorite quick tips when working with complicated queries. I don’t especially want to get involved an editor war, and offer the following only in the spirit of friendly cooperation for the Vim users out there.</p>
<p>As Luca mentioned, <a href="https://github.com/darold/pgFormatter">pgFormatter</a> is a convenient way to make SQL queries readable, automatically. It’s easy enough to feed it some SQL, and get a nice-looking result as output:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-sql" data-lang="sql"><span style="color:#a61717;background-color:#e3d2d2">$</span><span style="color:#bbb"> </span>pg_format<span style="color:#bbb"> </span><<span style="color:#bbb"> </span>create_outbreaks.<span style="color:#080;font-weight:bold">sql</span><span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#080;font-weight:bold">INSERT</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">INTO</span><span style="color:#bbb"> </span>outbreak<span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#080;font-weight:bold">SELECT</span><span style="color:#bbb">
</span><span style="color:#bbb"> </span>nextval(<span style="color:#d20;background-color:#fff0f0">'outbreak_id'</span>::regclass),<span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">extract</span>(<span style="color:#d20;background-color:#fff0f0">'year'</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">FROM</span><span style="color:#bbb"> </span>now())::<span style="color:#038">text</span><span style="color:#bbb"> </span>||<span style="color:#bbb"> </span><span style="color:#d20;background-color:#fff0f0">'-'</span><span style="color:#bbb"> </span>||<span style="color:#bbb"> </span>nextval(<span style="color:#d20;background-color:#fff0f0">'outbreak_number_seq'</span>)::<span style="color:#038">text</span>,<span style="color:#bbb"> </span><span style="color:#888">--number
</span><span style="color:#888"></span><span style="color:#bbb"> </span>(<span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">SELECT</span><span style="color:#bbb">
</span><span style="color:#bbb"> </span>first_name<span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">FROM</span><span style="color:#bbb"> </span>person<span style="color:#bbb"> </span>TABLESAMPLE<span style="color:#bbb"> </span>BERNOULLI<span style="color:#bbb"> </span>(<span style="color:#00d;font-weight:bold">10</span>)<span style="color:#bbb">
</span><span style="color:#bbb"></span><span style="color:#080;font-weight:bold">LIMIT</span><span style="color:#bbb"> </span><span style="color:#00d;font-weight:bold">1</span>),<span style="color:#bbb"> </span><span style="color:#888">-- name
</span><span style="color:#888"></span><span style="color:#bbb"> </span>NOW()<span style="color:#bbb"> </span>-<span style="color:#bbb"> </span><span style="color:#038">interval</span><span style="color:#bbb"> </span><span style="color:#d20;background-color:#fff0f0">'1 day'</span><span style="color:#bbb"> </span>*<span style="color:#bbb"> </span>random()<span style="color:#bbb"> </span>*<span style="color:#bbb"> </span><span style="color:#00d;font-weight:bold">100</span>,<span style="color:#bbb"> </span>(<span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">SELECT</span><span style="color:#bbb">
</span><span style="color:#bbb"> </span>id<span style="color:#bbb">
</span><span style="color:#bbb"> </span><span style="color:#080;font-weight:bold">FROM</span><span style="color:#bbb"> </span><span style="color:#d20;background-color:#fff0f0">"user"</span><span style="color:#bbb"> </span>TABLESAMPLE<span style="color:#bbb"> </span>BERNOULLI<span style="color:#bbb"> </span>(<span style="color:#00d;font-weight:bold">10</span>)<span style="color:#bbb">
</span><span style="color:#bbb"></span>...<span style="color:#bbb">
</span></code></pre></div><p>In my perfect world I might quibble with some of its formatting decisions, such as the lack of indent on the <code>LIMIT 1</code> line above. But in practice the results are good enough for my tastes that I haven’t bothered to investigate whether I can improve them. I just use it, and it’s good enough for me.</p>
<p>And because Vim lets me highlight a region, pipe it through an external program, and replace the region with that program’s output, it’s easy to use it simply by selecting a section of code and typing <code>:!pg_format</code> like this:</p>
<p><img src="/blog/2022/04/formatting-sql-vim-pgformat/sample.gif" width=250 height=492 alt="pgformatter example animation of terminal" /></p>
On Shapefiles and PostGIShttps://www.endpointdev.com/blog/2022/04/shapefiles-postgis/2022-04-02T00:00:00+00:00Josh Tolley
<p><img src="/blog/2022/04/shapefiles-postgis/endurance-clip.webp" alt="Partial map of the voyage of the Endurance, from the book “South”, Ernest H. Shackleton">
Partial map of the voyage of the Endurance, from <a href="https://www.gutenberg.org/ebooks/5199">“South”, by Ernest Shackleton</a></p>
<p>The shapefile format is commonly used in geospatial vector data interchange, but as it’s managed by a commercial entity, Esri, and as GIS is a fairly specialized field, and perhaps because the format specification is only <a href="https://en.wikipedia.org/wiki/Shapefile">“mostly open”</a>, these files can sometimes be confusing to the newcomer. Perhaps these notes can help clarify things.</p>
<p>Though the name “shapefile” would suggest a single file in filesystem parlance, a shapefile requires at least three different files, including filename extensions <code>.shp</code>, <code>.shx</code>, and <code>.dbf</code>, stored in the same directory, and the term “shapefile” often refers to that directory, or to an archive such as a zipfile or tarball containing that directory.</p>
<h3 id="qgis">QGIS</h3>
<p><a href="https://qgis.org">QGIS</a> is an open-source package to create, view, and process GIS data. One good first step with any shapefile, or indeed any GIS data, is often to take a look at it. Simply tell QGIS to open the shapefile directory. It may help to add other layers, such as one of the world map layers QGIS provides by default, to see the shapefile data in context.</p>
<h3 id="gdal">GDAL</h3>
<p>Though QGIS can convert between GIS formats itself, I prefer working in a command-line environment. <a href="https://gdal.org/">The GDAL software suite</a> aims to translate GIS data between many available formats, including shapefiles. I most commonly use its <code>ogr2ogr</code> command-line utility, along with the excellent accompanying manpage.</p>
<p>In short, a typical <code>ogr2ogr</code> command tells the utility where to find the input data and where to put the converted output, optionally with various reformatting and processing options. You’ll find some examples below.</p>
<h3 id="postgis">PostGIS</h3>
<p>Much of our (ok, my) GIS work has involved <a href="https://postgis.net">PostGIS</a>, an extension to the PostgreSQL database for handling GIS data. It’s been convenient for me to process GIS data using the same language and tools I use to process other data. It uses GDAL’s libraries internally.</p>
<h3 id="examples">Examples</h3>
<h4 id="import-shapefile-data-into-postgis">Import Shapefile data into PostGIS</h4>
<p>The example below comes from a customer’s project we recently worked on. They provided us a set of several shapefiles, which I first arranged in a directory structure. This code imports each of them into a PostGIS database, in the <code>shapefiles</code> schema.</p>
<p>The other arguments to <code>ogr2ogr</code> specify the output format (“PostgreSQL”), the destination database name, and the directory which stores the shapefile. <code>ogr2ogr</code> expects the destination and source arguments in that order, as two positional arguments, so here the destination is <code>PG:dbname=destdb</code>, and the source file name comes from the the <code>$i</code> script variable.</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash"><span style="color:#080;font-weight:bold">for</span> i in <span style="color:#d20;background-color:#fff0f0">`</span>find . -name <span style="color:#d20;background-color:#fff0f0">"*shp"</span><span style="color:#d20;background-color:#fff0f0">`</span>; <span style="color:#080;font-weight:bold">do</span>
<span style="color:#369">j</span>=<span style="color:#080;font-weight:bold">$(</span>basename <span style="color:#369">$i</span><span style="color:#080;font-weight:bold">)</span>
<span style="color:#369">k</span>=<span style="color:#33b;background-color:#fff0f0">${</span><span style="color:#369">j</span>/.shp/<span style="color:#33b;background-color:#fff0f0">}</span>
ogr2ogr -f PostgreSQL -nln shapefiles.<span style="color:#33b;background-color:#fff0f0">${</span><span style="color:#369">k</span><span style="color:#33b;background-color:#fff0f0">}</span> PG:dbname=destdb <span style="color:#369">$i</span>
<span style="color:#080;font-weight:bold">done</span>
</code></pre></div><h4 id="export-postgis-data-as-kml">Export PostGIS data as KML</h4>
<p>This example creates a KML file from PostGIS query results. The arguments provide the query to use to fetch the data, the output format (“KML”), the output file name, and the source database. This will create a KML file containing a set of unstyled placemarks, with names from the <code>property_code</code> column, and geometry data from the <code>outline_geom</code> column in the <code>properties</code> table of our database.</p>
<p>In this project, <code>outline_geom</code> contained GIS “linestrings”, data types consisting of a series of lines, which <code>ogr2ogr</code> translated into KML polygons. Had <code>outline_geom</code> contained points, for instance, the KML result would also have been points. In other words, <code>ogr2ogr</code> automatically chooses the correct KML object type based on the GIS object type in the input data.</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">ogr2ogr -sql <span style="color:#d20;background-color:#fff0f0">"select property_code, outline_geom from properties"</span> -f KML outlines.kml PG:dbname=properties
</code></pre></div><p>Note that though the examples above use PostGIS, <code>ogr2ogr</code> can take shapefile input and produce KML output directly without the PostGIS intermediary. We used PostGIS in these cases for other purposes, such as to filter the output and limit the attributes stored in the KML result.</p>
<p>By default, <code>ogr2ogr</code> puts all the attributes from the shapefile into <code>ExtendedData</code> elements in the KML, but in our case we didn’t want those. We also didn’t want all the entries in the shapefile in our resulting KML. To skip the PostGIS step, we might do something like this:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">ogr2ogr -f kml output.kml shapefile_directory/
</code></pre></div><p>What tools do you use for shapefile processing? Please let us know!</p>
Demonstrating the HotSwap JVMhttps://www.endpointdev.com/blog/2020/11/java-hotswap/2020-11-25T00:00:00+00:00Josh Tolley
<p><img src="/blog/2020/11/java-hotswap/zebras-scale-crush.jpg" alt="zebras"></p>
<p><a href="https://unsplash.com/photos/dgH8NSdEDv0">Photo</a> by <a href="https://unsplash.com/@valenciascott">Neil and Zulma Scott</a></p>
<p>For a recent Java development project I spent a while setting up an environment to take advantage of the HotSwap JVM, a Java virtual machine that automatically reloads classes when they change. This feature can potentially eliminate the need to redeploy each time code changes, reducing development cycle time considerably. While setting up the environment, I found I wanted a simple example of hot swapping available for my own experimentation, and I thought I’d share that example here.</p>
<p>First, let’s create a simple Java program. It needs to be slightly more complex than the ubiquitous “Hello, World!” application, because we need it to keep running for a while; if it just prints some message and exits immediately, we won’t have time to compile new code and see the hot swap feature in action. Here’s an example that uses a simple infinite loop, wherein it sleeps for one second, prints a message, and then repeats.</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-java" data-lang="java"><span style="color:#080;font-weight:bold">import</span> <span style="color:#b06;font-weight:bold">java.lang.Thread</span>;
<span style="color:#080;font-weight:bold">public</span> <span style="color:#080;font-weight:bold">class</span> <span style="color:#b06;font-weight:bold">HotSwapTest</span> {
<span style="color:#080;font-weight:bold">public</span> <span style="color:#080;font-weight:bold">static</span> <span style="color:#888;font-weight:bold">void</span> <span style="color:#06b;font-weight:bold">main</span>(String[] args) {
<span style="color:#080;font-weight:bold">while</span> (<span style="color:#080;font-weight:bold">true</span>) {
<span style="color:#080;font-weight:bold">try</span> {
System.<span style="color:#369">out</span>.<span style="color:#369">println</span>(<span style="color:#d20;background-color:#fff0f0">"Hi"</span>);
Thread.<span style="color:#369">sleep</span>(1000);
} <span style="color:#080;font-weight:bold">catch</span> (InterruptedException e) {
<span style="color:#888">// Ignore this
</span><span style="color:#888"></span> }
}
}
}
~
</code></pre></div><p>If I build this into build/classes/java/main and run it, as expected it prints out “Hi” every second:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">josh@igtre:~/hotswaptest$ java -cp build/classes/java/main/ HotSwapTest
Hi
Hi
Hi
...
</code></pre></div><p>The usual JVM doesn’t include the HotSwap feature. For my purposes I downloaded <a href="https://dcevm.github.io/">DCEVM</a>, an alternative JVM which includes HotSwap. It’s also possible to patch some existing JVMs to add HotSwap, if you’d prefer. When I run the same code with DCEVM, it runs the code just like it did with the normal JVM, with additional debugging output:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">Starting HotswapAgent '/home/josh/hotswaptest/dcevm/lib/hotswap/hotswap-agent.jar'
HOTSWAP AGENT: 15:01:23.423 INFO (org.hotswap.agent.HotswapAgent) - Loading Hotswap agent {1.4.1} - unlimited runtime class redefinition.
HOTSWAP AGENT: 15:01:24.189 INFO (org.hotswap.agent.config.PluginRegistry) - Discovered plugins: [JdkPlugin, Hotswapper, WatchResources, ClassInitPlugin, AnonymousClassPatch, Hibernate, Hibernate3JPA, Hibernate3, Spring, Jersey1, Jersey2, Jetty, Tomcat, ZK, Logback, Log4j2, MyFaces, Mojarra, Omnifaces, ELResolver, WildFlyELResolver, OsgiEquinox, Owb, Proxy, WebObjects, Weld, JBossModules, ResteasyRegistry, Deltaspike, GlassFish, Vaadin, Wicket, CxfJAXRS, FreeMarker, Undertow, MyBatis]
</code></pre></div><p>To make hot swapping work automatically, we need to provide the JVM with a properties file in the JVM’s classpath. Mine looks like this, and lives in build/classes/java/main, next to the compiled class files:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">autoHotswap=true
LOGGER=debug
</code></pre></div><p>These properties are pretty self-explanatory: they tell the JVM to hot swap automatically when it finds new code, and turn up logging to DEBUG level.</p>
<p>So, with that all set up, let’s run the program again, change the code and rebuild it, and see what happens. For this test, I’ll just edit the message printed in each loop from “Hi” to “Hello”.</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">josh@igtre:~/hotswaptest$ ./dcevm/bin/java -cp build/classes/java/main/ HotSwapTest
Starting HotswapAgent '/home/josh/hotswaptest/dcevm/lib/hotswap/hotswap-agent.jar'
HOTSWAP AGENT: 15:36:22.132 INFO (org.hotswap.agent.HotswapAgent) - Loading Hotswap agent {1.4.1} - unlimited runtime class redefinition.
HOTSWAP AGENT: 15:36:22.543 DEBUG (org.hotswap.agent.annotation.handler.OnClassLoadedHandler) - Init for method public static void org.hotswap.agent.plugin.jdk.JdkPlugin.flushIntrospectClassInfoCache(java.lang.ClassLoader,org.hotswap.agent.javassist.CtClass)
HOTSWAP AGENT: 15:36:22.546 DEBUG (org.hotswap.agent.util.HotswapTransformer) - Registering transformer for class regexp '.*'.
HOTSWAP AGENT: 15:36:22.549 DEBUG (org.hotswap.agent.annotation.handler.OnClassLoadedHandler) - Init for method public static void org.hotswap.agent.plugin.jdk.JdkPlugin.flushObjectStreamCaches(java.lang.ClassLoader,org.hotswap.agent.javassist.CtClass)
HOTSWAP AGENT: 15:36:22.550 DEBUG (org.hotswap.agent.util.HotswapTransformer) - Registering transformer for class regexp '.*'.
...
</code></pre></div><p>The flurry of DEBUG messages tells me that it must have read my properties file correctly, and when I change the code and rebuild, I see the JVM respond with still more debug messages, saying it found and reloaded my code:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">HOTSWAP AGENT: 15:38:27.066 DEBUG (org.hotswap.agent.watch.nio.WatcherNIO2) - Watch event 'ENTRY_DELETE' on '/home/josh/hotswaptest/build/classes/java/main/HotSwapTest.class' --> HotSwapTest.class
HOTSWAP AGENT: 15:38:27.118 DEBUG (org.hotswap.agent.watch.nio.WatcherNIO2) - Watch event 'ENTRY_CREATE' on '/home/josh/hotswaptest/build/classes/java/main/HotSwapTest.class' --> HotSwapTest.class
HOTSWAP AGENT: 15:38:27.119 DEBUG (org.hotswap.agent.watch.nio.WatcherNIO2) - Watch event 'ENTRY_MODIFY' on '/home/josh/hotswaptest/build/classes/java/main/HotSwapTest.class' --> HotSwapTest.class
HOTSWAP AGENT: 15:38:27.280 DEBUG (org.hotswap.agent.annotation.handler.WatchEventCommand) - Executing resource changed method watchReload on class org.hotswap.agent.plugin.hotswapper.HotswapperPlugin for event WatchFileEvent on path /home/josh/hotswaptest/build/classes/java/main/HotSwapTest.class for event ENTRY_MODIFY
...
HOTSWAP AGENT: 15:38:27.439 DEBUG (org.hotswap.agent.plugin.jdk.JdkPlugin) - Flushing HotSwapTest from introspector
Hi
HOTSWAP AGENT: 15:38:27.482 DEBUG (org.hotswap.agent.config.PluginManager) - ... reloaded classes [HotSwapTest] (autoHotswap)
Hi
Hi
Hi
</code></pre></div><p>But although it says it swapped in the new code successfully, it’s still printing “Hi”, not “Hello”. Why?</p>
<p>It turns out this ability to hot swap new code isn’t unlimited, and one important limitation is that methods you’re already running aren’t reloaded. Since the <code>main()</code> method was running, it didn’t get swapped out. What if I change the code so that instead of printing a string every second, it calls a method, and that method prints the string? Here’s some code to test that technique.</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-java" data-lang="java"><span style="color:#080;font-weight:bold">import</span> <span style="color:#b06;font-weight:bold">java.lang.Thread</span>;
<span style="color:#080;font-weight:bold">public</span> <span style="color:#080;font-weight:bold">class</span> <span style="color:#b06;font-weight:bold">HotSwapTest</span> {
<span style="color:#080;font-weight:bold">public</span> <span style="color:#080;font-weight:bold">static</span> <span style="color:#888;font-weight:bold">void</span> <span style="color:#06b;font-weight:bold">printMsg</span>() {
System.<span style="color:#369">out</span>.<span style="color:#369">println</span>(<span style="color:#d20;background-color:#fff0f0">"Here is printMsg"</span>);
}
<span style="color:#080;font-weight:bold">public</span> <span style="color:#080;font-weight:bold">static</span> <span style="color:#888;font-weight:bold">void</span> <span style="color:#06b;font-weight:bold">main</span>(String[] args) {
<span style="color:#080;font-weight:bold">while</span> (<span style="color:#080;font-weight:bold">true</span>) {
<span style="color:#080;font-weight:bold">try</span> {
HotSwapTest.<span style="color:#369">printMsg</span>();
Thread.<span style="color:#369">sleep</span>(1000);
} <span style="color:#080;font-weight:bold">catch</span> (InterruptedException e) {
<span style="color:#888">// Ignore this
</span><span style="color:#888"></span> }
}
}
}
</code></pre></div><p>Now, I start the JVM over again, and as expected, it prints “Here is printMsg” once every second. When I change to “Here is printMsg v2.0” and rebuild, this happens:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">Here is printMsg
Here is printMsg
HOTSWAP AGENT: 15:48:34.923 DEBUG (org.hotswap.agent.watch.nio.WatcherNIO2) - Watch event 'ENTRY_DELETE' on '/home/josh/hotswaptest/build/classes/java/main/HotSwapTest.class' --> HotSwapTest.class
HOTSWAP AGENT: 15:48:35.007 DEBUG (org.hotswap.agent.watch.nio.WatcherNIO2) - Watch event 'ENTRY_CREATE' on '/home/josh/hotswaptest/build/classes/java/main/HotSwapTest.class' --> HotSwapTest.class
HOTSWAP AGENT: 15:48:35.008 DEBUG (org.hotswap.agent.watch.nio.WatcherNIO2) - Watch event 'ENTRY_MODIFY' on '/home/josh/hotswaptest/build/classes/java/main/HotSwapTest.class' --> HotSwapTest.class
HOTSWAP AGENT: 15:48:35.174 DEBUG (org.hotswap.agent.annotation.handler.WatchEventCommand) - Executing resource changed method watchReload on class org.hotswap.agent.plugin.hotswapper.HotswapperPlugin for event WatchFileEvent on path /home/josh/hotswaptest/build/classes/java/main/HotSwapTest.class for event ENTRY_CREATE
HOTSWAP AGENT: 15:48:35.188 DEBUG (org.hotswap.agent.plugin.hotswapper.HotswapperPlugin) - Class HotSwapTest will be reloaded from URL file:/home/josh/hotswaptest/build/classes/java/main/HotSwapTest.class
HOTSWAP AGENT: 15:48:35.175 DEBUG (org.hotswap.agent.annotation.handler.WatchEventCommand) - Executing resource changed method watchReload on class org.hotswap.agent.plugin.hotswapper.HotswapperPlugin for event WatchFileEvent on path /home/josh/hotswaptest/build/classes/java/main/HotSwapTest.class for event ENTRY_MODIFY
HOTSWAP AGENT: 15:48:35.191 DEBUG (org.hotswap.agent.plugin.hotswapper.HotswapperPlugin) - Class HotSwapTest will be reloaded from URL file:/home/josh/hotswaptest/build/classes/java/main/HotSwapTest.class
HOTSWAP AGENT: 15:48:35.352 DEBUG (org.hotswap.agent.command.impl.SchedulerImpl) - Executing pluginManager.hotswap([class HotSwapTest])
HOTSWAP AGENT: 15:48:35.355 RELOAD (org.hotswap.agent.config.PluginManager) - Reloading classes [HotSwapTest] (autoHotswap)
HOTSWAP AGENT: 15:48:35.370 DEBUG (org.hotswap.agent.plugin.jdk.JdkPlugin) - Flushing HotSwapTest from com.sun.beans.introspect.ClassInfo cache
HOTSWAP AGENT: 15:48:35.375 DEBUG (org.hotswap.agent.plugin.jdk.JdkPlugin) - Flushing HotSwapTest from ObjectStreamClass caches
HOTSWAP AGENT: 15:48:35.376 DEBUG (org.hotswap.agent.plugin.jdk.JdkPlugin) - Flushing HotSwapTest from introspector
HOTSWAP AGENT: 15:48:35.415 DEBUG (org.hotswap.agent.config.PluginManager) - ... reloaded classes [HotSwapTest] (autoHotswap)
Here is printMsg v2.0
Here is printMsg v2.0
</code></pre></div><p>As you can see, it swapped in the new code correctly, and now prints the new message. HotSwap was a success!</p>
<p>I imagine there are very few production environments where this feature would be applicable, and even in development, getting this to work properly for something like a JEE app deployed to some application container isn’t necessarily a simple task. But if it can cut down on redeployment cycles, it can certainly be a valuable developer tool.</p>
What is SharePoint?https://www.endpointdev.com/blog/2020/03/what-is-sharepoint/2020-03-25T00:00:00+00:00Dan Briones
<p><img src="/blog/2020/03/what-is-sharepoint/servers.jpg" alt="Web servers"></p>
<p><a href="https://unsplash.com/photos/M5tzZtFCOfs">Image</a> by <a href="https://unsplash.com/@tvick">Taylor Vick</a></p>
<p>People often ask me about SharePoint, Microsoft’s browser-based collaboration platform which allows users to upload and share all kinds of documents, images, messages, and more. The product has nearly two decades of history and there are still many who don’t know much about it.</p>
<p>The SharePoint platform has grown over those years, but its capabilities have expanded in such a way that it can be quickly dismissed from consideration out of fear of the complexity of its implementation and the cost of deployment. These fears may be unfounded, however. Especially if you are already on Office 365, SharePoint may be included in your plan.</p>
<p>SharePoint was designed as a framework to create and share content on the web without the need to write code. Its purpose was to allow everyone in the organization to collaborate without any specific programming skills. This framework grew over time, adding many different types of content allowing for interactions with other frameworks increasing the effectiveness of any organization’s work product or intellectual property and communications.</p>
<h3 id="flavors-of-sharepoint">Flavors of SharePoint</h3>
<p>There are two ‘flavors’ of SharePoint. You can use Microsoft’s cloud-based service or you can host your own on-premises server farm. But I suspect Microsoft’s preference is to wrangle organizations into the cloud, as seen in Microsoft’s SharePoint 2019 online documentation which casually omits references to the on-premises server product. Microsoft offers an inexpensive per-user SharePoint cloud service license for those organizations that don’t want to use Office 365’s other offerings.</p>
<p>On the other hand, on-premises SharePoint Server licensing is very expensive, especially if you wish to design for high availability and create a well-balanced SharePoint server farm. This requires CALs (Client Access Licenses) as well. But the cloud licensing model is very attractive in pricing, especially if you are planning to move your organization’s Exchange email into the Office 365 offering because SharePoint licensing is included in the top two Business tiers and all the Enterprise licensing plans.</p>
<h3 id="intranets">Intranets</h3>
<p>Over the years I have helped many small- to medium-sized businesses create their intranets using both on-prem SharePoint servers and the SharePoint Online offering, mostly to leverage document management features and their content search capabilities. SharePoint is very good at indexing all Microsoft Office formats, data and metadata, allowing for the inclusion of custom extended tags that can be applied to files and folders to further categorize them and make them easy to organize and find. It also indexes content and metadata from readable PDF format.</p>
<p>Because the environment is highly customizable or “brandable”, companies quickly expand on its use once they are introduced to its basic capabilities. I’m often surprised by how creative non-technical staff can be as they come up with new ways to use the platform.</p>
<p>SharePoint is also a secure way to share documents on a variety of devices including mobile, via the web, leveraging Active Directory or SAML compliant single-sign on (SSO) services like Okta, OneLogin, or Duo for authentication. The framework has its own content permission group capabilities that are simple to manage without giving content-managers access to AD or auth servers. This framework is attractive because it provides, without much training, the ability for employees to create and share content with granular permissions, manage data and custom lists, and create individual web pages or entire sites within the Portal.</p>
<h3 id="sharepoint-sites">SharePoint Sites</h3>
<p>Let’s discuss SharePoint Sites. SharePoint allows users with permissions to create individual pages or entire “Team Sites” to organize and secure content with permissions that the site creator can define. The ability to create content and assign these permissions within the organization or with external partners or customers can be delegated by an administrator to team leaders who wish to control their own content. However, global administrators retain the control to secure the company’s data and intellectual property via built-in tools, policies, auditing, and alerts. There is also a reporting system for compliance reporting.</p>
<p>Sites can contain an assortment of content types. Team managers can simply pick from a list of available components, including document and photo libraries, and custom lists that are created much like Excel tables to hold data for distribution, such as employee directories, product lists, and inventory items. Sites can also contain other shared resources such as Wikis, calendars, tasks, issue tracking, and OneNote notebooks. Sites can contain components you can create yourself with Visual Studio leveraging the API. Each site can also hold its own set of usage statistics and workflow management for teams to optimize and hone both performance and effectiveness of the data shared. In my experience sales, finance, and HR teams benefit the most from these ready-made components, but all manner of teams can find useful tools on this platform.</p>
<h3 id="power-apps-power-automate-flow">Power Apps, Power Automate (Flow)</h3>
<p>Each SharePoint release adds to its predecessor’s core functionality. One such example is “Power Apps”, an add-on service that allows using external data sources capable of interacting with the built-in lists and library components without coding, by establishing connectors and forming mobile content pages.</p>
<p>Another called “Power Automate” (formerly known as “Flow”) can reduce repetitive tasks by using a simple visual designer for scripting actions that respond to triggered events. This varied set of tools also integrates seamlessly with not just the base Office products like Word, Excel, and PowerPoint, but also with Teams and OneNote and even other major services like Exchange and Dynamics. This increases the collaboration options for a mobile workforce regardless of the device or OS they use. This is a very powerful tool for organizations needing to collaborate across different platforms and around the world.</p>
<h3 id="mobile">Mobile</h3>
<p>There are currently native mobile apps that are free on iOS and Android for mobile access to SharePoint content. This provides an additional layer of security and also compartmentalizing for Mobile Device Management (MDM) systems that run business content in a separate and encrypted container for work on these devices. This may include systems like IBM’s Maas360, SOTI MobiControl, Citrix XenMobile, AirWatch by VMware or Microsoft’s own Microsoft Intune.</p>
<p>With all these out-of-the-box capabilities, SharePoint is perfect for intranet portals, group sites that can be created and managed without any programming experience or knowledge. But if you have the skill set, the entire framework and all its capabilities are created upon a well-documented and time-tested API, with which a person can easily expand with web components that can be created via Visual Studio, Microsoft’s own IDE (Integrated Development Environment), and in several programming languages, allowing integration with other Microsoft API frameworks like MS SQL Server, Exchange, and Dynamics.</p>
<h3 id="keeping-up-with-content-changes">Keeping up with content changes</h3>
<p>All this functionality and capability sometimes intimidates some businesses. It would seem that you could easily overwhelm your users. To combat that, the whole system, at every level, allows users to “follow” content. This is functionality that has been in the platform since inception and is one of my favorite features. For every page, every site or component at any level, users have the ability to subscribe to content. You can follow with one click any document and be notified in your portal homepage or via notifications of actions taken or content changed. This means that you can allow people to be as connected as they feel they need to be. Nobody needs to suffer from information overload or out-of-control mobile notifications.</p>
<h3 id="we-can-help">We can help!</h3>
<p>If you are already a Microsoft Office 365 subscriber, odds are you already have access to this incredible tool without any additional cost. How can you leverage this for your business? Visit the following content, or <a href="/contact/">contact us</a> at End Point for additional information.</p>
<ul>
<li><a href="https://techcommunity.microsoft.com/t5/sharepoint/ct-p/SharePoint">The SharePoint Community</a></li>
<li><a href="https://developer.microsoft.com/en-us/sharepoint">SharePoint Developer Network</a></li>
<li><a href="https://support.microsoft.com/en-us/office/sign-in-to-sharepoint-324a89ec-e77b-4475-b64a-13a0c14c45ec?ui=en-us&rs=en-us&ad=us">SharePoint online training</a></li>
</ul>
Useful terminal toolshttps://www.endpointdev.com/blog/2020/01/useful-terminal-tools/2020-01-03T00:00:00+00:00Jon Jensen
<p><img src="/blog/2020/01/useful-terminal-tools/41825658781_b9f7f79fc1_o-crop.jpg" alt="Móricz Zsigmond körtér Underground Station (people, escalators)" /><br><a href="https://www.flickr.com/photos/tcee35mm/41825658781/">Photo by Tee Cee</a> · <a href="https://creativecommons.org/licenses/by/2.0/">CC BY 2.0</a>, cropped</p>
<p>Like most of my co-workers, I spend a lot of time in a terminal emulator (console) in a shell at the Linux command line. I often come across tools that make work there nicer, but sometimes I forget about them before I integrate them into my workflow. So here are notes about a few of them for myself and anyone else who may find them useful.</p>
<h3 id="httpie">HTTPie</h3>
<p><a href="https://httpie.io/">HTTPie</a> is:</p>
<blockquote>
<p>a command line HTTP client with an intuitive UI, JSON support, syntax highlighting, wget-like downloads, plugins, and more.</p>
</blockquote>
<p>Given how commonly-used curl, wget, and GET/POST (lwp-request) are, it is nice to see some innovation in this space to enhance usability.</p>
<p>Here is a simple example that demonstrates several HTTP redirects with full request and response headers, colorized:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">http -v --pretty=all --follow endpointdev.com | less -R
</code></pre></div><p>The color highlighting of the body, not just response headers, is the main difference here from curl, wget, etc.</p>
<p>Also nice for ad-hoc interactive use is that the verbose header output is sent to <code>stdout</code> instead of <code>stderr</code>, so it shows up in <code>less</code> without needing to have the shell merge it with <code>2>&1</code> before piping to <code>less</code>.</p>
<h4 id="an-aside-on-http-redirects">An aside on HTTP redirects</h4>
<p>In the above example, the client makes 3 requests, because the first 2 are redirects:</p>
<ul>
<li><a href="http://endpointdev.com/">http://endpointdev.com/</a></li>
<li><a href="https://endpointdev.com/">https://endpointdev.com/</a></li>
<li><a href="https://www.endpointdev.com/">https://www.endpointdev.com/</a></li>
</ul>
<p>Normally we would want to reduce the number of HTTP redirects, so why not redirect straight from <code>http://endpointdev.com/</code> to <code>https://www.endpointdev.com/</code>?</p>
<p>Before the introduction of <a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Strict-Transport-Security">HTTP Strict Transport Security</a> (HSTS) to the web, that is what we did.</p>
<p>But with HSTS it is better to pass through HTTPS for each hostname, so that the <code>Strict-Transport-Security</code> HTTP response header can be sent and the browser can cache the fact that both the bare yourdomain.tld and <a href="http://www.yourdomain.tld">www.yourdomain.tld</a> should only be accessed via HTTPS.</p>
<p>See the thorough description at the <a href="https://www.sentinelstand.com/article/http-strict-transport-security-hsts-canonical-www-redirects">Sentinel Stand blog post</a> for more details, including a discussion of <code>includeSubDomains</code> traps.</p>
<h4 id="json-pretty-printing">JSON pretty-printing</h4>
<p>HTTPie also can pretty-print JSON responses. For example, compare this <a href="https://mdn.github.io/learning-area/javascript/oojs/json/superheroes.json">sample JSON from MDN</a> raw from <code>curl</code>:</p>
<code class="hljs">
% curl -i /blog/2020/01/useful-terminal-tools/super-hero-squad.json<br>
HTTP/1.1 200 OK<br>
<span style="font-weight:bold;">Date</span>: Tue, 14 Jan 2020 01:48:58 GMT<br>
<span style="font-weight:bold;">Server</span>: Apache<br>
<span style="font-weight:bold;">Last-Modified</span>: Tue, 23 Apr 2019 00:43:10 GMT<br>
<span style="font-weight:bold;">ETag</span>: "22a-58727de80a380"<br>
<span style="font-weight:bold;">Accept-Ranges</span>: bytes<br>
<span style="font-weight:bold;">Content-Length</span>: 554<br>
<span style="font-weight:bold;">Content-Type</span>: application/json<br>
<br>
{"squadName":"Super hero squad","homeTown":"Metro City","formed":2016,"secretBase":"Super tower","active":true,"members":[{"name":"Molecule Man","age":29,"secretIdentity":"Dan Jukes","powers":["Radiation resistance","Turning tiny","Radiation blast"]},{"name":"Madame Uppercut","age":39,"secretIdentity":"Jane Wilson","powers":["Million tonne punch","Damage resistance","Superhuman reflexes"]},{"name":"Eternal Flame","age":1000000,"secretIdentity":"Unknown","powers":["Immortality","Heat Immunity","Inferno","Teleportation","Interdimensional travel"]}]}
</code><p></p>
<p>to HTTPie’s pretty-printed output:</p>
<code class="hljs">
% http /blog/2020/01/useful-terminal-tools/super-hero-squad.json<br>
<span style="color: rgb(0,135,255);">HTTP</span><span style="color:white;"></span><span style="color: rgb(138,138,138);">/</span><span style="color:white;"></span><span style="color: rgb(0,175,175);">1.1</span><span style="color:white;"></span><span style="color: rgb(138,138,138);"> </span><span style="color:white;"></span><span style="color: rgb(0,175,175);">200</span><span style="color:white;"></span><span style="color: rgb(138,138,138);"> </span><span style="color:white;"></span><span style="color: rgb(175,135,0);">OK</span><span style="color:white;"><br>
</span><span style="color: rgb(138,138,138);">Accept-Ranges</span><span style="color:white;"></span><span style="color: rgb(138,138,138);">:</span><span style="color:white;"></span><span style="color: rgb(138,138,138);"> </span><span style="color:white;"></span><span style="color: rgb(0,175,175);">bytes</span><span style="color:white;"><br>
</span><span style="color: rgb(138,138,138);">Connection</span><span style="color:white;"></span><span style="color: rgb(138,138,138);">:</span><span style="color:white;"></span><span style="color: rgb(138,138,138);"> </span><span style="color:white;"></span><span style="color: rgb(0,175,175);">Keep-Alive</span><span style="color:white;"><br>
</span><span style="color: rgb(138,138,138);">Content-Length</span><span style="color:white;"></span><span style="color: rgb(138,138,138);">:</span><span style="color:white;"></span><span style="color: rgb(138,138,138);"> </span><span style="color:white;"></span><span style="color: rgb(0,175,175);">554</span><span style="color:white;"><br>
</span><span style="color: rgb(138,138,138);">Content-Type</span><span style="color:white;"></span><span style="color: rgb(138,138,138);">:</span><span style="color:white;"></span><span style="color: rgb(138,138,138);"> </span><span style="color:white;"></span><span style="color: rgb(0,175,175);">application/json</span><span style="color:white;"><br>
</span><span style="color: rgb(138,138,138);">Date</span><span style="color:white;"></span><span style="color: rgb(138,138,138);">:</span><span style="color:white;"></span><span style="color: rgb(138,138,138);"> </span><span style="color:white;"></span><span style="color: rgb(0,175,175);">Tue, 14 Jan 2020 02:27:29 GMT</span><span style="color:white;"><br>
</span><span style="color: rgb(138,138,138);">ETag</span><span style="color:white;"></span><span style="color: rgb(138,138,138);">:</span><span style="color:white;"></span><span style="color: rgb(138,138,138);"> </span><span style="color:white;"></span><span style="color: rgb(0,175,175);">"22a-58727de80a380"</span><span style="color:white;"><br>
</span><span style="color: rgb(138,138,138);">Keep-Alive</span><span style="color:white;"></span><span style="color: rgb(138,138,138);">:</span><span style="color:white;"></span><span style="color: rgb(138,138,138);"> </span><span style="color:white;"></span><span style="color: rgb(0,175,175);">timeout=5, max=100</span><span style="color:white;"><br>
</span><span style="color: rgb(138,138,138);">Last-Modified</span><span style="color:white;"></span><span style="color: rgb(138,138,138);">:</span><span style="color:white;"></span><span style="color: rgb(138,138,138);"> </span><span style="color:white;"></span><span style="color: rgb(0,175,175);">Tue, 23 Apr 2019 00:43:10 GMT</span><span style="color:white;"><br>
</span><span style="color: rgb(138,138,138);">Server</span><span style="color:white;"></span><span style="color: rgb(138,138,138);">:</span><span style="color:white;"></span><span style="color: rgb(138,138,138);"> </span><span style="color:white;"></span><span style="color: rgb(0,175,175);">Apache</span><span style="color:white;"><br>
<br>
</span><span style="color: rgb(138,138,138);">{</span><span style="color:white;"><br>
</span><span style="color: rgb(138,138,138);"> </span><span style="color:white;"></span><span style="color: rgb(0,135,255);">"active"</span><span style="color:white;"></span><span style="color: rgb(138,138,138);">:</span><span style="color:white;"></span><span style="color: rgb(138,138,138);"> </span><span style="color:white;"></span><span style="color: rgb(215,95,0);">true</span><span style="color:white;"></span><span style="color: rgb(138,138,138);">,</span><span style="color:white;"><br>
</span><span style="color: rgb(138,138,138);"> </span><span style="color:white;"></span><span style="color: rgb(0,135,255);">"formed"</span><span style="color:white;"></span><span style="color: rgb(138,138,138);">:</span><span style="color:white;"></span><span style="color: rgb(138,138,138);"> </span><span style="color:white;"></span><span style="color: rgb(0,175,175);">2016</span><span style="color:white;"></span><span style="color: rgb(138,138,138);">,</span><span style="color:white;"><br>
</span><span style="color: rgb(138,138,138);"> </span><span style="color:white;"></span><span style="color: rgb(0,135,255);">"homeTown"</span><span style="color:white;"></span><span style="color: rgb(138,138,138);">:</span><span style="color:white;"></span><span style="color: rgb(138,138,138);"> </span><span style="color:white;"></span><span style="color: rgb(0,175,175);">"Metro City"</span><span style="color:white;"></span><span style="color: rgb(138,138,138);">,</span><span style="color:white;"><br>
</span><span style="color: rgb(138,138,138);"> </span><span style="color:white;"></span><span style="color: rgb(0,135,255);">"members"</span><span style="color:white;"></span><span style="color: rgb(138,138,138);">:</span><span style="color:white;"></span><span style="color: rgb(138,138,138);"> </span><span style="color:white;"></span><span style="color: rgb(138,138,138);">[</span><span style="color:white;"><br>
</span><span style="color: rgb(138,138,138);"> </span><span style="color:white;"></span><span style="color: rgb(138,138,138);">{</span><span style="color:white;"><br>
</span><span style="color: rgb(138,138,138);"> </span><span style="color:white;"></span><span style="color: rgb(0,135,255);">"age"</span><span style="color:white;"></span><span style="color: rgb(138,138,138);">:</span><span style="color:white;"></span><span style="color: rgb(138,138,138);"> </span><span style="color:white;"></span><span style="color: rgb(0,175,175);">29</span><span style="color:white;"></span><span style="color: rgb(138,138,138);">,</span><span style="color:white;"><br>
</span><span style="color: rgb(138,138,138);"> </span><span style="color:white;"></span><span style="color: rgb(0,135,255);">"name"</span><span style="color:white;"></span><span style="color: rgb(138,138,138);">:</span><span style="color:white;"></span><span style="color: rgb(138,138,138);"> </span><span style="color:white;"></span><span style="color: rgb(0,175,175);">"Molecule Man"</span><span style="color:white;"></span><span style="color: rgb(138,138,138);">,</span><span style="color:white;"><br>
</span><span style="color: rgb(138,138,138);"> </span><span style="color:white;"></span><span style="color: rgb(0,135,255);">"powers"</span><span style="color:white;"></span><span style="color: rgb(138,138,138);">:</span><span style="color:white;"></span><span style="color: rgb(138,138,138);"> </span><span style="color:white;"></span><span style="color: rgb(138,138,138);">[</span><span style="color:white;"><br>
</span><span style="color: rgb(138,138,138);"> </span><span style="color:white;"></span><span style="color: rgb(0,175,175);">"Radiation resistance"</span><span style="color:white;"></span><span style="color: rgb(138,138,138);">,</span><span style="color:white;"><br>
</span><span style="color: rgb(138,138,138);"> </span><span style="color:white;"></span><span style="color: rgb(0,175,175);">"Turning tiny"</span><span style="color:white;"></span><span style="color: rgb(138,138,138);">,</span><span style="color:white;"><br>
</span><span style="color: rgb(138,138,138);"> </span><span style="color:white;"></span><span style="color: rgb(0,175,175);">"Radiation blast"</span><span style="color:white;"><br>
</span><span style="color: rgb(138,138,138);"> </span><span style="color:white;"></span><span style="color: rgb(138,138,138);">]</span><span style="color:white;"></span><span style="color: rgb(138,138,138);">,</span><span style="color:white;"><br>
</span><span style="color: rgb(138,138,138);"> </span><span style="color:white;"></span><span style="color: rgb(0,135,255);">"secretIdentity"</span><span style="color:white;"></span><span style="color: rgb(138,138,138);">:</span><span style="color:white;"></span><span style="color: rgb(138,138,138);"> </span><span style="color:white;"></span><span style="color: rgb(0,175,175);">"Dan Jukes"</span><span style="color:white;"><br>
</span><span style="color: rgb(138,138,138);"> </span><span style="color:white;"></span><span style="color: rgb(138,138,138);">}</span><span style="color:white;"></span><span style="color: rgb(138,138,138);">,</span><span style="color:white;"><br>
</span><span style="color: rgb(138,138,138);"> </span><span style="color:white;"></span><span style="color: rgb(138,138,138);">{</span><span style="color:white;"><br>
</span><span style="color: rgb(138,138,138);"> </span><span style="color:white;"></span><span style="color: rgb(0,135,255);">"age"</span><span style="color:white;"></span><span style="color: rgb(138,138,138);">:</span><span style="color:white;"></span><span style="color: rgb(138,138,138);"> </span><span style="color:white;"></span><span style="color: rgb(0,175,175);">39</span><span style="color:white;"></span><span style="color: rgb(138,138,138);">,</span><span style="color:white;"><br>
</span><span style="color: rgb(138,138,138);"> </span><span style="color:white;"></span><span style="color: rgb(0,135,255);">"name"</span><span style="color:white;"></span><span style="color: rgb(138,138,138);">:</span><span style="color:white;"></span><span style="color: rgb(138,138,138);"> </span><span style="color:white;"></span><span style="color: rgb(0,175,175);">"Madame Uppercut"</span><span style="color:white;"></span><span style="color: rgb(138,138,138);">,</span><span style="color:white;"><br>
</span><span style="color: rgb(138,138,138);"> </span><span style="color:white;"></span><span style="color: rgb(0,135,255);">"powers"</span><span style="color:white;"></span><span style="color: rgb(138,138,138);">:</span><span style="color:white;"></span><span style="color: rgb(138,138,138);"> </span><span style="color:white;"></span><span style="color: rgb(138,138,138);">[</span><span style="color:white;"><br>
</span><span style="color: rgb(138,138,138);"> </span><span style="color:white;"></span><span style="color: rgb(0,175,175);">"Million tonne punch"</span><span style="color:white;"></span><span style="color: rgb(138,138,138);">,</span><span style="color:white;"><br>
</span><span style="color: rgb(138,138,138);"> </span><span style="color:white;"></span><span style="color: rgb(0,175,175);">"Damage resistance"</span><span style="color:white;"></span><span style="color: rgb(138,138,138);">,</span><span style="color:white;"><br>
</span><span style="color: rgb(138,138,138);"> </span><span style="color:white;"></span><span style="color: rgb(0,175,175);">"Superhuman reflexes"</span><span style="color:white;"><br>
</span><span style="color: rgb(138,138,138);"> </span><span style="color:white;"></span><span style="color: rgb(138,138,138);">]</span><span style="color:white;"></span><span style="color: rgb(138,138,138);">,</span><span style="color:white;"><br>
</span><span style="color: rgb(138,138,138);"> </span><span style="color:white;"></span><span style="color: rgb(0,135,255);">"secretIdentity"</span><span style="color:white;"></span><span style="color: rgb(138,138,138);">:</span><span style="color:white;"></span><span style="color: rgb(138,138,138);"> </span><span style="color:white;"></span><span style="color: rgb(0,175,175);">"Jane Wilson"</span><span style="color:white;"><br>
</span><span style="color: rgb(138,138,138);"> </span><span style="color:white;"></span><span style="color: rgb(138,138,138);">}</span><span style="color:white;"></span><span style="color: rgb(138,138,138);">,</span><span style="color:white;"><br>
</span><span style="color: rgb(138,138,138);"> </span><span style="color:white;"></span><span style="color: rgb(138,138,138);">{</span><span style="color:white;"><br>
</span><span style="color: rgb(138,138,138);"> </span><span style="color:white;"></span><span style="color: rgb(0,135,255);">"age"</span><span style="color:white;"></span><span style="color: rgb(138,138,138);">:</span><span style="color:white;"></span><span style="color: rgb(138,138,138);"> </span><span style="color:white;"></span><span style="color: rgb(0,175,175);">1000000</span><span style="color:white;"></span><span style="color: rgb(138,138,138);">,</span><span style="color:white;"><br>
</span><span style="color: rgb(138,138,138);"> </span><span style="color:white;"></span><span style="color: rgb(0,135,255);">"name"</span><span style="color:white;"></span><span style="color: rgb(138,138,138);">:</span><span style="color:white;"></span><span style="color: rgb(138,138,138);"> </span><span style="color:white;"></span><span style="color: rgb(0,175,175);">"Eternal Flame"</span><span style="color:white;"></span><span style="color: rgb(138,138,138);">,</span><span style="color:white;"><br>
</span><span style="color: rgb(138,138,138);"> </span><span style="color:white;"></span><span style="color: rgb(0,135,255);">"powers"</span><span style="color:white;"></span><span style="color: rgb(138,138,138);">:</span><span style="color:white;"></span><span style="color: rgb(138,138,138);"> </span><span style="color:white;"></span><span style="color: rgb(138,138,138);">[</span><span style="color:white;"><br>
</span><span style="color: rgb(138,138,138);"> </span><span style="color:white;"></span><span style="color: rgb(0,175,175);">"Immortality"</span><span style="color:white;"></span><span style="color: rgb(138,138,138);">,</span><span style="color:white;"><br>
</span><span style="color: rgb(138,138,138);"> </span><span style="color:white;"></span><span style="color: rgb(0,175,175);">"Heat Immunity"</span><span style="color:white;"></span><span style="color: rgb(138,138,138);">,</span><span style="color:white;"><br>
</span><span style="color: rgb(138,138,138);"> </span><span style="color:white;"></span><span style="color: rgb(0,175,175);">"Inferno"</span><span style="color:white;"></span><span style="color: rgb(138,138,138);">,</span><span style="color:white;"><br>
</span><span style="color: rgb(138,138,138);"> </span><span style="color:white;"></span><span style="color: rgb(0,175,175);">"Teleportation"</span><span style="color:white;"></span><span style="color: rgb(138,138,138);">,</span><span style="color:white;"><br>
</span><span style="color: rgb(138,138,138);"> </span><span style="color:white;"></span><span style="color: rgb(0,175,175);">"Interdimensional travel"</span><span style="color:white;"><br>
</span><span style="color: rgb(138,138,138);"> </span><span style="color:white;"></span><span style="color: rgb(138,138,138);">]</span><span style="color:white;"></span><span style="color: rgb(138,138,138);">,</span><span style="color:white;"><br>
</span><span style="color: rgb(138,138,138);"> </span><span style="color:white;"></span><span style="color: rgb(0,135,255);">"secretIdentity"</span><span style="color:white;"></span><span style="color: rgb(138,138,138);">:</span><span style="color:white;"></span><span style="color: rgb(138,138,138);"> </span><span style="color:white;"></span><span style="color: rgb(0,175,175);">"Unknown"</span><span style="color:white;"><br>
</span><span style="color: rgb(138,138,138);"> </span><span style="color:white;"></span><span style="color: rgb(138,138,138);">}</span><span style="color:white;"><br>
</span><span style="color: rgb(138,138,138);"> </span><span style="color:white;"></span><span style="color: rgb(138,138,138);">]</span><span style="color:white;"></span><span style="color: rgb(138,138,138);">,</span><span style="color:white;"><br>
</span><span style="color: rgb(138,138,138);"> </span><span style="color:white;"></span><span style="color: rgb(0,135,255);">"secretBase"</span><span style="color:white;"></span><span style="color: rgb(138,138,138);">:</span><span style="color:white;"></span><span style="color: rgb(138,138,138);"> </span><span style="color:white;"></span><span style="color: rgb(0,175,175);">"Super tower"</span><span style="color:white;"></span><span style="color: rgb(138,138,138);">,</span><span style="color:white;"><br>
</span><span style="color: rgb(138,138,138);"> </span><span style="color:white;"></span><span style="color: rgb(0,135,255);">"squadName"</span><span style="color:white;"></span><span style="color: rgb(138,138,138);">:</span><span style="color:white;"></span><span style="color: rgb(138,138,138);"> </span><span style="color:white;"></span><span style="color: rgb(0,175,175);">"Super hero squad"</span><span style="color:white;"><br>
</span><span style="color: rgb(138,138,138);">}</span>
</code><p></p>
<p>Of course you can use something like the popular <code>jq</code> to do your pretty-printing, but getting it in one tool with no options is nice.</p>
<p>Perhaps the hardest thing for me to remember with HTTPie is the program name, simply <code>http</code>. I forget and type <code>httpie</code> and get nothing. But no, I won’t give in and create an alias or symlink to it with that name. It is shorter and the default and I need to learn it!</p>
<h3 id="htty">htty</h3>
<p>Related, but on the opposite end of the user interface spectrum, is <a href="https://htty.github.io/htty/">htty, the HTTP TTY</a>. It has its own shell and state that you work with. The examples give a good feel for it.</p>
<p>Keeping the context and history in a console interface is really nice for interactive work compared to the one-shot nature of curl, wget, etc.</p>
<h3 id="exporting-ansi-terminal-colors">Exporting ANSI terminal colors</h3>
<p>Many command-line tools can now output colors, bold and underlined type, etc. for interactive user readability. But how do we convert those escape codes to HTML so we can show them on the Web, as I did above?</p>
<p><a href="https://github.com/theZiz/aha">aha (Ansi HTML Adapter)</a> (C), <a href="https://github.com/pycontribs/ansi2html">ansi2html</a> (Python), and <a href="https://github.com/pixelb/scripts/blob/master/scripts/ansi2html.sh">ansi2html.sh</a> (Bourne shell) are 3 different ways to accomplish that.</p>
<p>If you have any trouble getting a program to output ANSI color codes to a file (since many of them refuse to do that when the output is not an interactive terminal), you can use <code>script</code>:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">script -q -c <span style="color:#d20;background-color:#fff0f0">"/the/command/here with options"</span> /path/to/output
</code></pre></div><p>to capture it verbatim. Since <code>script</code> is part of the standard <code>util-linux</code> package, you probably already have it on any Linux system.</p>
<h3 id="git-browsing">Git browsing</h3>
<p>These two programs are really nice Git repository browsers for the terminal:</p>
<ul>
<li><a href="https://jonas.github.io/tig/">tig (Text-mode Interface for Git)</a></li>
<li><a href="https://github.com/rgburke/grv">grv (Git Repository Viewer)</a></li>
</ul>
<p>Just go to their project pages and look at the screenshots and try them out on a repository of your own. Experiencing that is better than what I could generically show you. They are great for browsing branches in complex merge histories, files, etc.</p>
<h3 id="introductory-manual-pages">Introductory manual pages</h3>
<p>You are probably aware that the shorthand “TL;DR” means “too long; didn’t read”. There is now a project called <a href="https://tldr.sh/">TLDR pages</a> whic is “a community effort to simplify the beloved man pages with practical examples”.</p>
<p>I have been trying to get used to the newer <code>ss</code> (which stands for “socket statistics”) which can replace <code>netstat</code> and <code>lsof</code>, so I took a look at what the TLDR pages had to say about it:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">% tldr ss
ss
Utility to investigate sockets.
- Show all TCP/UDP/RAW/UNIX sockets:
ss -a -t|-u|-w|-x
- Filter TCP sockets by states, only/exclude:
ss state/exclude bucket/big/connected/synchronized/...
- Show all TCP sockets connected to the local HTTPS port (443):
ss -t src :443
- Show all TCP sockets along with processes connected to a remote ssh port:
ss -pt dst :ssh
- Show all UDP sockets connected on specific source and destination ports:
ss -u 'sport == :source_port and dport == :destination_port'
- Show all TCP IPv4 sockets locally connected on the subnet 192.168.0.0/16:
ss -4t src 192.168/16
</code></pre></div><p>That gets me started a lot faster than the ~350 lines that <code>man ss</code> gives me. And the full details are always still there for me when I need them.</p>
<p>There are TLDR pages on many commands!</p>
<h3 id="ping-graph">Ping graph</h3>
<p><a href="https://github.com/orf/gping">Check out gping</a>, which makes a textual graph of ping responses. Simple and handy.</p>
<h3 id="finding-unicode-characters">Finding Unicode characters</h3>
<p>There is a nice command-line utility for dealing with Unicode in various ways. Ricardo Signes explains why he wrote it and demonstrates how to use it in <a href="https://rjbs.manxome.org/rubric/entry/2061">“I rewrote uni”</a>.</p>
<p>Install with <code>cpanm App::Uni</code> or however else you prefer to install Perl modules from CPAN. It has a few useful options:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">% uni
usage:
uni SEARCH-TERMS... - find codepoints with matching names or values
uni [-s] ONE-CHARACTER - print the codepoint and name of one character
uni -n SEARCH-TERMS... - find codepoints with matching names
uni -c STRINGS... - print out the codepoints in a string
uni -u CODEPOINTS... - look up and print hex codepoints
Other switches:
-8 - also show the UTF-8 bytes to encode
</code></pre></div><p>If you see a Unicode character out in the wild and can copy and paste it, <code>uni</code> can identify it for you:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">% uni 🦆
🦆 - U+1F986 - DUCK
</code></pre></div><p>I most commonly use its implicit search function:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">% uni horse
⻢- U+02EE2 - CJK RADICAL C-SIMPLIFIED HORSE
⾺- U+02FBA - KANGXI RADICAL HORSE
🎠 - U+1F3A0 - CAROUSEL HORSE
🏇 - U+1F3C7 - HORSE RACING
🐎 - U+1F40E - HORSE
🐴 - U+1F434 - HORSE FACE
🝖 - U+1F756 - ALCHEMICAL SYMBOL FOR HORSE DUNG
🩣 - U+1FA63 - XIANGQI RED HORSE
🩪 - U+1FA6A - XIANGQI BLACK HORSE
</code></pre></div><p>That Unicode character 🝖 = “alchemical symbol for horse dung” is a delight. I know I will be using that one often! 😜</p>
<p><code>uni</code> is also useful for showing the characters for given code points:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">% uni -u 25c8 25c9 25ca
◈ - U+025C8 - WHITE DIAMOND CONTAINING BLACK SMALL DIAMOND
◉ - U+025C9 - FISHEYE
◊ - U+025CA - LOZENGE
</code></pre></div><p>There is also an unrelated <a href="https://github.com/arp242/uni">Go version of <code>uni</code></a> that is fairly similar:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">% uni help
Usage: uni [-hrq] [help | identify | search | print | emoji]
Flags:
-q Quiet output; don't print header, "no matches", etc.
-r "Raw" output instead of displaying graphical variants for control
characters and ◌ (U+25CC) before combining characters.
Commands:
identify [string string ...]
Idenfity all the characters in the given strings.
search [word word ...]
Search description for any of the words.
print [ident ident ...]
Print characters by codepoint, category, or block:
Codepoints U+2042, U+2042..U+2050
Categories and Blocks OtherPunctuation, Po, GeneralPunctuation
all Everything
Names are matched case insensitive; spaces and commas are optional and
can be replaced with an underscore. "Po", "po", "punction, OTHER",
"Punctuation_other", and PunctuationOther are all identical.
emoji [-tone tone,..] [-gender gender,..] [-groups word] [word word ...]
Search emojis. The special keyword "all" prints all emojis.
-group comma-separated list of group and/or subgroup names.
-tone comma-separated list of light, mediumlight, medium,
mediumdark, dark. Default is to include none.
-gender comma-separated list of person, man, or woman.
Default is to include all.
Note: output may contain unprintable character (U+200D and U+FE0F) which
may not survive a select and copy operation from text-based applications
such as terminals. It's recommended to copy to the clipboard directly
with e.g. xclip.
</code></pre></div><p>Its search function is roughly the same as its Perl counterpart with the <code>-8</code> option and additionally showing the decimal code point (which I’ve never seen used) and the HTML hex entity:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">% uni search horse
cpoint dec utf-8 html name
'⻢' U+2EE2 12002 e2 bb a2 &#x2ee2; CJK RADICAL C-SIMPLIFIED HORSE (Other_Symbol)
'⾺' U+2FBA 12218 e2 be ba &#x2fba; KANGXI RADICAL HORSE (Other_Symbol)
'🎠' U+1F3A0 127904 f0 9f 8e a0 &#x1f3a0; CAROUSEL HORSE (Other_Symbol)
'🏇' U+1F3C7 127943 f0 9f 8f 87 &#x1f3c7; HORSE RACING (Other_Symbol)
'🐎' U+1F40E 128014 f0 9f 90 8e &#x1f40e; HORSE (Other_Symbol)
'🐴' U+1F434 128052 f0 9f 90 b4 &#x1f434; HORSE FACE (Other_Symbol)
'🝖' U+1F756 128854 f0 9f 9d 96 &#x1f756; ALCHEMICAL SYMBOL FOR HORSE DUNG (Other_Symbol)
'🩣' U+1FA63 129635 f0 9f a9 a3 &#x1fa63; XIANGQI RED HORSE (Other_Symbol)
'🩪' U+1FA6A 129642 f0 9f a9 aa &#x1fa6a; XIANGQI BLACK HORSE (Other_Symbol)
</code></pre></div><h3 id="dashboard">Dashboard</h3>
<p>The <a href="https://wtfutil.com/">WTF terminal dashboard</a> looks pretty neat, but at the moment, it feels like too much. I may come back to it later.</p>
<h3 id="chromecast">Chromecast</h3>
<p>I never knew I want a way to cast video and audio to my Chromecast, but with <a href="https://github.com/skorokithakis/catt/#cast-all-the-things">catt (Cast All The Things)</a> that is what we get!</p>
<p>It works well for me. Sadly, Google discontinued manufacturing the Chromecast Audio in 2019. Ours still works and I hope it continues to for a long time to come.</p>
<h3 id="mega-meta-list-of-tools">Mega-meta list of tools</h3>
<p>Finally, if the above has not given you enough new toys to play with, see <a href="https://github.com/trimstray/the-book-of-secret-knowledge">The Book of Secret Knowledge</a>, “A collection of inspiring lists, manuals, cheatsheets, blogs, hacks, one-liners, CLI/web tools, and more”.</p>
<p>Some of its list is links to other lists. I’ll see you in a few years when you get through all that!</p>
A Project Manager’s Toolkithttps://www.endpointdev.com/blog/2019/11/project-managers-toolkit/2019-11-16T00:00:00+00:00Elizabeth Garrett Christensen
<p><img src="/blog/2019/11/project-managers-toolkit/banner.jpg" alt=""></p>
<p>I do a lot of project management work related to web applications. Everyone assembles their own set of useful productivity tools and I’m feeling pretty good about how I’ve got things set up at the moment so I thought I’d share my secret sauce. I’m interested to hear what other people have found useful as well so feel free to share in the comments.</p>
<div>
<img src="/blog/2019/11/project-managers-toolkit/slack-icon.png" style="margin: 15px; height: 80px" />
<img src="/blog/2019/11/project-managers-toolkit/zulip-icon.png" style="margin: 15px; height: 80px" />
<img src="/blog/2019/11/project-managers-toolkit/skype-icon.png" style="margin: 15px; height: 80px" />
<img src="/blog/2019/11/project-managers-toolkit/hangouts-icon.png" style="margin: 15px; height: 80px" />
</div>
<h3 id="chat-tools">Chat Tools</h3>
<p>Ok, this one is all over the map. <a href="https://slack.com/">Slack</a> is obviously the big one and many of our clients use that with us. Most larger organizations incur a cost (smaller projects are free). Internally End Point uses <a href="https://zulipchat.com/">Zulip</a>, which is a pretty decent open source application. Zulip doesn’t have as refined a user interface as Slack, but it works for our needs, and also has a mobile app I can use on the go.</p>
<p>I also highly recommend <a href="https://www.skype.com/en/">Skype</a>. While it’s not the best system out there, so many people use it, it is almost impossible to avoid. Skype is really easy way to get someone’s attention and I use it with a lot of my clients for quick one-on-one questions and chats. <a href="https://hangouts.google.com/">Google Hangouts</a> chat is also really nice for working with external folks and will run in your menu tray and mobile phone, so you can stay up to date wherever you’re working.</p>
<div>
<img src="/blog/2019/11/project-managers-toolkit/zoom-logo.png" style="margin: 15px; height: 40px" />
<img src="/blog/2019/11/project-managers-toolkit/meet-icon.png" style="margin: 15px; height: 80px" />
</div>
<h3 id="video-calls">Video Calls</h3>
<p>I spend a few hours on the phone a day and my go-to system at the moment is <a href="https://zoom.us/">Zoom</a>. End Point often uses Google Hangouts or Meet, but those require a Google account, which can be a burden for some clients or contacts who aren’t Google-ified. Zoom is pretty cost effective and lets you connect with someone via video chat or phone and even provides local numbers for your participants all over the globe. This is not only a great system for large group meetings, it is excellent for initial meetings or situations where you’re unsure if your group needs a phone or video call. Each participant can choose for themselves. It’s also technically the most stable and is far better about maintaining solid connection (for my OS/browser) when compared to Skype or Google Meet/Hangouts. I also really like the ‘My Meeting’ area on Zoom, which allows you to start a meeting with anyone at anytime with just a link.</p>
<p>I should also note that I use Skype as my main office phone number, which lets me both accept calls in with a regular phone number and call out. Their annual plan of ~$40 per year is competitive with all of the other VoIP systems, and since I’m already using it for chats and video calls the desktop and mobile app integrates many of my communication channels.</p>
<div>
<img src="/blog/2019/11/project-managers-toolkit/chrome-icon.png" style="margin: 15px; height: 80px" />
<img src="/blog/2019/11/project-managers-toolkit/screencastify-logo.png" style="margin: 15px; height: 50px" />
</div>
<h3 id="screen-capture">Screen Capture</h3>
<p>One thing I’ve used recently that’s a nice communication tool is screen and voice capture. This lets me report bugs or describe functionality to a developer team really easily. I can just record my voice on top of a screen share. This has some of the same benefits as a video call, but I can do it when it’s convenient for me, and my recipient can listen to it whenever it’s convenient for them. This is also a good way to demo new features for clients that aren’t able to attend weekly standup meetings, but want to see progress. I’ve been using <a href="https://www.screencastify.com/">Screencastify</a> for this—it’s a free Google Chrome extension. It’s super easy to use and lets you send a Google Drive link to your video.</p>
<div>
<img src="/blog/2019/11/project-managers-toolkit/bugzilla-logo.png" style="margin: 15px; height: 50px" />
<img src="/blog/2019/11/project-managers-toolkit/rt-logo.jpg" style="margin: 15px; height: 70px" />
<img src="/blog/2019/11/project-managers-toolkit/redmine-logo.png" style="margin: 15px; height: 50px" />
<img src="/blog/2019/11/project-managers-toolkit/trello-icon.png" style="margin: 15px; height: 80px" />
</div>
<h3 id="ticket-trackers">Ticket Trackers</h3>
<p>There are lots of people trying to get your project management dollars with tracking systems. I am old school and liked <a href="https://www.bugzilla.org/">Bugzilla</a>, <a href="https://bestpractical.com/request-tracker">RT</a>, <a href="https://www.redmine.org/">Redmine</a>, and the tracking systems of 10 years ago. These are generally open source and self-hosted.</p>
<p>Nowadays, Atlassian has taken over the ticket tracking world with their software-as-a-service offering <a href="https://www.atlassian.com/software/jira">JIRA</a>. JIRA is nice but it is an extra expense that some projects want to avoid. I also find JIRA to be more complicated and hard to use for clients that aren’t used to these kinds of tools.</p>
<p><a href="https://basecamp.com/">Basecamp</a> has some great stuff too.</p>
<p>The one tool that really seems to work for almost all my client projects is <a href="https://trello.com">Trello</a>. It’s free, it’s easy to use for new people, and it is just so simple that it’s hard to lose anything or miscommunicate. I have probably 90% of my projects in Trello at the moment.</p>
<div>
<img src="/blog/2019/11/project-managers-toolkit/moqups-logo.png" style="margin: 15px; height: 70px" />
<img src="/blog/2019/11/project-managers-toolkit/draw-io-logo.png" style="margin: 15px; height: 80px" />
</div>
<h3 id="drawing-tools">Drawing tools</h3>
<p>I do quite a bit of wireframing and little drawings of features, webpages, workflows, and diagrams. I really like using <a href="https://moqups.com/">moqups</a>. This tool works really fast for me, especially for quick mockups of web pages and mobile apps. It has a lot of pre-formatted tables, form entries, and other common web page elements that make wireframing really fast. You can also quickly download drawings to PDF, which is great for chats or emails. You can also use the team feature for working in a collaboration.</p>
<p>I also really like the <a href="https://www.draw.io/">draw.io</a> plugin for Google Drive. For me, this is better for workflow charts and other diagrams and drawings. It can be especially great for doing screen share drawing or projects where you already are using a Google Drive to collect project docs.</p>
<div>
<img src="/blog/2019/11/project-managers-toolkit/google-calendar-icon.png" style="margin: 15px; height: 80px" />
<img src="/blog/2019/11/project-managers-toolkit/cloudhq-logo.png" style="margin: 15px; height: 50px" />
</div>
<h3 id="calendaring">Calendaring</h3>
<p>Since End Point is already using <a href="https://gsuite.google.com">G Suite</a> (the paid version of Google’s services), Google Calendar is a must-have. If you work with teams around the world, I recommend adding in Google’s pre-existing calendars by country. This makes it really easy to plan for holidays and festivals with your remote team. I’ve currently got holiday calendars for India, Poland, Eastern Europe, and the US. I think it’s also fun to talk about foreign holidays staff are taking off so I’m learning a little about other cultures as I go.</p>
<p>I’ve recently discovered that the plugin <a href="https://support.cloudhq.net/how-to-schedule-sending-of-email/">CloudHQ</a> will let you insert a button or link into emails so someone can view your open calendar time and schedule time with you, without them seeing your actual calendar. This has been really useful when trying to find time for a meeting and you want to give someone a lot of options. I’ve combined my work and home Google Calendars (yes, for my 5 kids) and they’re all color coded. Cloud HQ will show times in there that aren’t blocked out for other stuff.</p>
<div>
<img src="/blog/2019/11/project-managers-toolkit/bluestacks-logo.png" style="margin: 15px; height: 80px" />
<img src="/blog/2019/11/project-managers-toolkit/browserstack-logo.png" style="margin: 15px; height: 60px" />
</div>
<h3 id="phone-apps">Phone Apps</h3>
<p>I do work with a client on Android development and I really like a system called <a href="https://www.bluestacks.com/">BlueStacks</a>, which is an emulator for testing Android apps on a PC. Testing phone apps on a PC is much easier for just getting work done without having to pull up and install new apps on a separate device. Bluestacks is freeware and loaded with ads like most free apps are these days. Despite that, it is really stable and holds up well to a bit of pounding. It’s obviously geared towards gamers but their product works for my purpose too.</p>
<p>Testing iOS apps is easier for me, since I own several iOS devices and always have an iPhone or iPad at arm’s reach. One of my recent work hacks is to use the Zoom app to do a screen share <em>from</em> my phone. I just used this recently to discuss some mobile phone design issues on a website with a group of clients and developers, and it worked really well for showing the mobile site and talking through the changes needed.</p>
<p>Testing web applications in lots of different browsers and screen sizes for any design project is a huge priority and I’ve really liked using <a href="https://www.browserstack.com/">BrowserStack</a> for this. You download a local copy of the application, and can run any website in any OS/browser/screen size combo you can dream up. End Point lets our clients determine the browsers they want to support on their project. Smaller budgets usually equals less browser support, larger budgets generally cast a wider net in the browser world. The decision about which browsers we support is something decided earlier on the project usually after doing some research with site analytics and customer base. BrowserStack will work for projects where you’re supporting only a few main browsers or tons of them. It’s also really nice for testing your app in an older browser like IE 11. Let’s face it, no one wants to run a system that supports IE 11, and I’m happy to let BrowserStack help me with this. Their base cost is $39 per month and they let you start and stop your membership based on your testing needs.</p>
<h3 id="your-turn">Your turn!</h3>
<p>What are your favorite tools? I’d love to hear about things that help you get your job done, save time, and keep things organized.</p>
Winning with CSS Grid and Autoprefixerhttps://www.endpointdev.com/blog/2019/11/winning-with-css-grid-and-autoprefixer/2019-11-04T00:00:00+00:00Greg Davidson
<p><img src="/blog/2019/11/winning-with-css-grid-and-autoprefixer/grid.jpg" alt="Aerial view of St. Vitus Cathedral, Prague"></p>
<p>Photo by <a href="https://unsplash.com/photos/BWtyq5fn6Ng">Stijn te Strake</a> on <a href="https://unsplash.com/">Unsplash</a></p>
<h3 id="using-css-grid-today">Using CSS Grid Today</h3>
<p><a href="https://caniuse.com/#search=css%20grid">Support for CSS Grid</a> is excellent (over 90% globally! 💯). I have been using it in a variety of client projects for the past few years with great success. The combination of CSS Grid and Flexbox has made layout on the web much simpler, more fun and less frustrating compared to the <del>hacks</del> techniques which were used for CSS layout in the past. Definitely give it a try if you haven’t already done so.</p>
<p>IE10 and IE11 support an earlier version of the CSS Grid spec and with help from Autoprefixer we can reap the benefits of CSS Grid in those old browsers with a little bit of extra work. I’ve long been a huge fan and happy user of <a href="https://github.com/postcss/autoprefixer">Autoprefixer</a>. It’s a <a href="https://postcss.org/">PostCSS</a> plugin that examines your CSS and augments it with <a href="https://developer.mozilla.org/en-US/docs/Glossary/Vendor_Prefix">vendor prefixes</a> and alternative syntaxes. Although vendor prefixes (e.g. <code>-webkit</code>, <code>-moz</code>, <code>-o</code>, etc.) are much less common these days, there are a few still in use.</p>
<h3 id="css-grid-basics">CSS Grid Basics</h3>
<p>With CSS Grid the basic idea is to define the grid, i.e. the rows and columns that make it up, and then go about placing elements on the grid. Think of a website layout with a masthead, navigation, hero image, main content sidebar, and footer as an example. Your grid styles can specify where each of these elements should be displayed on the grid: how many rows or columns they span, etc. In addition to this method of explicitly placing elements on the grid, <a href="https://developer.mozilla.org/en-US/docs/Web/CSS/CSS_Grid_Layout/Auto-placement_in_CSS_Grid_Layout">auto-placement</a> places elements on the grid for you automatically.</p>
<h3 id="autoprefixer--css-grid">Autoprefixer + CSS Grid</h3>
<p>I recently learned how to use <a href="https://github.com/postcss/autoprefixer#control-comments">control comments in Autoprefixer</a> to get fine-grained control over the way it operates on and enhances my CSS Grid styles. Rather than specifying a single (global) behavior via config options, control comments let you tell Autoprefixer how you would like it to process individual blocks of CSS code. For the scenario where I was explicitly placing elements on my grid I used the following control comment in my CSS file:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-css" data-lang="css"><span style="color:#888">/* autoprefixer grid: no-autoplace */</span>
</code></pre></div><p>This allowed me to lay out the elements and Autoprefixer generated the <code>-ms-grid</code> styles required to make my modern CSS Grid styles work in IE 10 and IE 11.</p>
<p>In another scenario I <em><strong>did</strong></em> want the auto-placement support and enabled it with the following control comment in a different block of the same CSS file:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-css" data-lang="css"><span style="color:#888">/* autoprefixer grid: autoplace */</span>
</code></pre></div><p>This was for a group of uniformly sized tile elements that I wanted to lay out in a grid that adapted as the viewport size varied. E.g.: from 4 tiles per row, to 3 tiles, to 2 and finally to 1 tile per row for the smallest viewports.</p>
<p>In both cases I was pleasantly surprised that my CSS Grid styles worked in modern browsers <strong>and</strong> also in IE 11 and 10! I added some fallback styles using flexbox for those browsers which do not yet support CSS Grid to ensure the content was accessible there as well.</p>
<h3 id="test-your-autoprefixd-grid-styles">Test Your Autoprefix’d Grid Styles</h3>
<p>Please note the Autoprefixer docs warn its grid support may not work in <em>all</em> cases and must be tested. For my purposes this worked you great but make sure to test because your mileage may vary.</p>
<h3 id="learning-more">Learning More</h3>
<p>If you’d like to learn more about CSS Grid I would recommend checking out <a href="https://jensimmons.com/">Jen Simmons’</a> <a href="https://www.youtube.com/playlist?list=PLbSquHt1VCf0b43dfLKTrCriXdlZcmgoi">CSS Grid Basics</a> series on YouTube and the <a href="https://gridbyexample.com/">Grid by Example</a> site by <a href="https://rachelandrew.co.uk/">Rachel Andrew</a>. To learn more about Autoprefixer, check out <a href="https://github.com/postcss/autoprefixer">the docs</a> at GitHub or you can <a href="https://autoprefixer.github.io/">try it out interactively</a>.</p>
Linting Ruby In Your Editorhttps://www.endpointdev.com/blog/2019/06/linting-ruby-in-your-editor/2019-06-27T00:00:00+00:00Patrick Lewis
<p><img src="/blog/2019/06/linting-ruby-in-your-editor/banner.jpg" alt="Cotton" /> <a href="https://flic.kr/p/azENYB">Photo</a> by <a href="https://www.flickr.com/people/kimberlykv/">Kimberly Vardeman</a>, used under <a href="https://creativecommons.org/licenses/by/2.0/">CC BY 2.0</a></p>
<p>Ruby developers have access to a variety of <a href="https://en.wikipedia.org/wiki/Lint_(software)">linters</a> and <a href="https://en.wikipedia.org/wiki/Static_program_analysis">static program analysis</a> tools that can greatly improve developer efficiency and code quality by catching syntax errors, detecting <a href="https://en.wikipedia.org/wiki/Code_smell">code smells</a>, and making coding style suggestions based on popular style guides.</p>
<p>I have been a long-time advocate of configuring development environments with automatic code linting and will use this post to highlight some of the available tools for Ruby and methods for integrating them with popular code editors (Vim, Emacs, and Visual Studio Code).</p>
<p>Configuring your editor for automatic linting makes it much easier to identify and fix issues with your code at development time, and in-editor integration is very convenient for highlighting problems as you type (or save), making it easy to evaluate and improve the quality of your code as it is written.</p>
<p>Three popular linting plugins/extensions for Vim, Visual Studio Code, and Emacs are:</p>
<h3 id="asynchronous-lint-engine-alehttpsgithubcomw0rpale-plugin-for-vim"><a href="https://github.com/w0rp/ale">Asynchronous Lint Engine (ALE)</a> Plugin for Vim</h3>
<p>Provides asynchronous linting in Vim while you edit, and displays warnings/error messages in the editor. Supports the following tools for Ruby development and runs them automatically if they are found in your PATH:</p>
<ul>
<li><a href="https://brakemanscanner.org">brakeman</a></li>
<li><a href="https://github.com/flyerhzm/rails_best_practices">rails_best_practices</a></li>
<li><a href="https://github.com/troessner/reek">reek</a></li>
<li><a href="https://github.com/rubocop-hq/rubocop">rubocop</a></li>
<li><a href="https://www.ruby-lang.org/en/">ruby -wc</a> (verbose syntax check)</li>
<li><a href="https://github.com/ruby-formatter/rufo">rufo</a></li>
<li><a href="https://solargraph.org">solargraph</a></li>
<li><a href="https://github.com/testdouble/standard">standardrb</a></li>
</ul>
<h3 id="rubyhttpsmarketplacevisualstudiocomitemsitemnamerebornixrubylinters-extension-for-visual-studio-code"><a href="https://marketplace.visualstudio.com/items?itemName=rebornix.Ruby#linters">Ruby</a> Extension for Visual Studio Code</h3>
<p>Requires configuring the settings JSON file to enable each tool on an individual basis:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-json" data-lang="json"><span style="color:#a61717;background-color:#e3d2d2">//</span> <span style="color:#a61717;background-color:#e3d2d2">Basic</span> <span style="color:#a61717;background-color:#e3d2d2">settings:</span> <span style="color:#a61717;background-color:#e3d2d2">turn</span> <span style="color:#a61717;background-color:#e3d2d2">linter(s)</span> <span style="color:#a61717;background-color:#e3d2d2">on</span>
<span style="color:#d20;background-color:#fff0f0">"ruby.lint"</span><span style="color:#a61717;background-color:#e3d2d2">:</span> {
<span style="color:#b06;font-weight:bold">"reek"</span>: <span style="color:#080;font-weight:bold">true</span>,
<span style="color:#b06;font-weight:bold">"rubocop"</span>: <span style="color:#080;font-weight:bold">true</span>,
<span style="color:#b06;font-weight:bold">"ruby"</span>: <span style="color:#080;font-weight:bold">true</span>, <span style="color:#a61717;background-color:#e3d2d2">//Runs</span> <span style="color:#a61717;background-color:#e3d2d2">ruby</span> <span style="color:#a61717;background-color:#e3d2d2">-wc</span>
<span style="color:#b06;font-weight:bold">"fasterer"</span>: <span style="color:#080;font-weight:bold">true</span>,
<span style="color:#b06;font-weight:bold">"debride"</span>: <span style="color:#080;font-weight:bold">true</span>,
<span style="color:#b06;font-weight:bold">"ruby-lint"</span>: <span style="color:#080;font-weight:bold">true</span>
}<span style="color:#a61717;background-color:#e3d2d2">,</span>
</code></pre></div><ul>
<li><a href="https://github.com/seattlerb/debride">debride</a></li>
<li><a href="https://github.com/DamirSvrtan/fasterer">fasterer</a></li>
<li><a href="https://github.com/troessner/reek">reek</a></li>
<li><a href="https://github.com/rubocop-hq/rubocop">rubocop</a></li>
<li><a href="https://www.ruby-lang.org/en/">ruby -wc</a> (verbose syntax check)</li>
<li><a href="https://gitlab.com/yorickpeterse/ruby-lint">ruby-lint</a> (unmaintained)</li>
</ul>
<h3 id="flycheckhttpswwwflycheckorgenlatest-extension-for-emacs"><a href="https://www.flycheck.org/en/latest/">Flycheck</a> Extension for Emacs</h3>
<p>Detects and uses the following tools when editing Ruby code:</p>
<ul>
<li><a href="https://github.com/troessner/reek">reek</a></li>
<li><a href="https://github.com/rubocop-hq/rubocop">rubocop</a></li>
<li><a href="https://www.ruby-lang.org/en/">ruby -wc</a> (verbose syntax check)</li>
<li><a href="https://gitlab.com/yorickpeterse/ruby-lint">ruby-lint</a> (unmaintained)</li>
</ul>
<h3 id="linter-selection">Linter Selection</h3>
<p>I suggest starting small and only installing one or two linters to begin with; some choices provide similar features and will display conflicting or redundant warnings/errors if used at the same time. I think that <a href="https://github.com/rubocop-hq/rubocop">RuboCop</a> is a great first choice, and I have spent years using it as my primary linter, though recently I have started supplementing it with <a href="https://github.com/troessner/reek">Reek</a> and <a href="https://github.com/DamirSvrtan/fasterer">Fasterer</a>.</p>
<p>I highly recommend the use of these tools and consider them essential for any Ruby developer that is interested in improving the quality, reliability, and maintainability of their code.</p>
Vue.js Remote Devtools Reviewhttps://www.endpointdev.com/blog/2019/06/vue-remote-devtools-review/2019-06-01T00:00:00+00:00Patrick Lewis
<p><img src="/blog/2019/06/vue-remote-devtools-review/banner.jpg" alt="Wrenches" /> <a href="https://flic.kr/p/DsF7MA">Photo</a> by <a href="https://www.flickr.com/photos/30478819@N08/">Marco Verch</a>, used under <a href="https://creativecommons.org/licenses/by/2.0/">CC BY 2.0</a></p>
<p>Most Vue.js developers will be familiar with <a href="https://github.com/vuejs/vue-devtools">Vue.js devtools</a> in the form of <a href="https://chrome.google.com/webstore/detail/vuejs-devtools/nhdogjmejiglipccpnnnanhbledajbpd">Chrome</a> or <a href="https://addons.mozilla.org/en-US/firefox/addon/vue-js-devtools/">Firefox</a> browser extensions/addons. The Vue.js devtools integrate nicely into Chrome and Firefox’s native developer tools in the form of a new ‘Vue’ tab that provides a developer with the ability to browse their component hierarchy, investigate the state of their application’s Vuex store, and several other useful features.</p>
<p>I was a longtime <a href="https://www.apple.com/safari/">Safari</a> user who eventually became disappointed with its limited extension support (as compared to Chrome or Firefox-based browsers); there was once a Safari version of Vue.js devtools that required some manual installation, but development on that version <a href="https://github.com/vuejs/vue-devtools/issues/632#issuecomment-373657010">ended back in early 2018</a>.</p>
<p>An alternative for Safari users, or developers who want to debug Vue.js applications running on other clients such as mobile devices, is the standalone <a href="https://github.com/vuejs/vue-devtools/blob/master/shells/electron/README.md">vue-remote-devtools</a> app. vue-remote-devtools is an <a href="https://electronjs.org/">Electron</a> app that runs in its own window and is loaded in a Vue.js application via a remote connection. I was curious to see how the developer experience of using vue-remote-devools compared to the Chrome browser extension that I was familiar with.</p>
<p>Installation of vue-remote-devtools was simple, with just a single <code>yarn global add @vue/devtools</code> command. Once installed, I was able to run the app with a <code>vue-devtools</code> command which opened a new window on my desktop:</p>
<p><img src="/blog/2019/06/vue-remote-devtools-review/vue-devtools-0.png" alt="Vue.js remote devtools window" /></p>
<p>Then it was a matter of updating my Vue.js application to connect to Vue.js remote devtools running locally on my laptop. I added the <code><script src="http://localhost:8098"></script></code> tag to my application’s index page as described in the vue-remote-devtools documentation. When I loaded my Vue.js application in Safari I was confronted with a new problem: a <a href="https://content-security-policy.com/">Content-Security-Policy</a> error preventing the browser from connecting to local port 8098:</p>
<p><img src="/blog/2019/06/vue-remote-devtools-review/vue-devtools-1.png" alt="Browser console error" /></p>
<p>My Vue.js application is served from a Rails application via <a href="https://github.com/rails/webpacker">webpacker</a> and already had a CSP in place, so it was relatively easy to update that with the needed configuration:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-ruby" data-lang="ruby"><span style="color:#888"># config/initializers/content_security_policy.rb</span>
<span style="color:#036;font-weight:bold">Rails</span>.application.config.content_security_policy <span style="color:#080;font-weight:bold">do</span> |policy|
policy.script_src <span style="color:#a60;background-color:#fff0f0">:self</span>, <span style="color:#a60;background-color:#fff0f0">:https</span>, <span style="color:#a60;background-color:#fff0f0">:unsafe_eval</span>, <span style="color:#a60;background-color:#fff0f0">:unsafe_inline</span>, <span style="color:#d20;background-color:#fff0f0">'http://localhost:8098'</span>
<span style="color:#080;font-weight:bold">end</span>
</code></pre></div><p>After restarting Rails to load the updated CSP initializer, my Vue.js application successfully connected to Vue.js remote devtools and I saw an interface that was almost identical to what I was used to from the Chrome Vue.js devtools extension:</p>
<p><img src="/blog/2019/06/vue-remote-devtools-review/vue-devtools-2.png" alt="Vue.js remote devtools window" /></p>
<p>I’m glad that vue-remote-devtools exists and provides feature parity with the browser-based devtools, especially for debugging Vue.js applications running on mobile devices where there are no alternatives, but I will be sticking with the Chrome extension for my everyday use. I didn’t like the additional configuration needed to add a separate script tag and make a change to the CSP just to get the remote devtools connection established; in a multi-developer project, the script tag would introduce “Failed to load resource” browser console errors for any other developers who didn’t happen to be running remote devtools on their machines. I also found myself missing the way that the Chrome/Firefox extensions integrate nicely alongside those browsers’ existing developer tools, making it easy to switch between the Vue.js devtools and the JavaScript console or Elements view.</p>
Roundup of some useful websiteshttps://www.endpointdev.com/blog/2018/12/roundup-of-some-useful-websites/2018-12-21T00:00:00+00:00Jon Jensen
<p><a href="/blog/2018/12/roundup-of-some-useful-websites/squoosh-demo-20181220b.png"><img src="/blog/2018/12/roundup-of-some-useful-websites/squoosh-demo-20181220a.jpg" /></a></p>
<p>The world is a big place, and the Internet has gotten pretty big too. There are always new projects being created, and I want to share some useful and interesting ones from my growing list:</p>
<h3 id="squoosh-image-compressor">Squoosh image compressor</h3>
<p>Squoosh, hosted at <a href="https://squoosh.app/">squoosh.app</a>, is an open source in-browser tool for experimenting with image compression, made by the Chrome development team.</p>
<p>With Squoosh you can load an image in your browser, convert it to different image file formats (JPEG, WebP, PNG, BMP) using various compression algorithms and settings, and compare the result side-by-side with either the original image or the image compressed using other options.</p>
<p>The screenshot above demonstrates Squoosh running in Firefox 64 on Linux. Click on it to see a larger, lossless PNG screenshot. The photo was taken by my son Phin in northern Virginia, and is a typical imperfect mobile phone photo. On the left is the original, and on the right I am showing how bad gradients in the sky can look when compressed too much—maybe a quality level of 12 (out of 100) was too low. It does make for a very compact file size, though. 😄</p>
<p>Squoosh’s interface has a convenient slider bar so you can compare any part of the two versions of the image side by side. You can zoom and pan the image as well.</p>
<p>It is neat to see JavaScript tools (in this case TypeScript specifically) doing work in the browser that has traditionally been done by native apps.</p>
<h3 id="nerd-fonts">Nerd Fonts</h3>
<p>If you want access to an amazing number of symbols in a font, check out <a href="https://nerdfonts.com/">nerdfonts.com</a>. There you can mix and match symbols from many popular developer-oriented fonts such as Font Awesome, Powerline Symbols, Material Design, etc.</p>
<p>I probably should have chosen some fun symbols to demonstrate it here, but I could tell that was a rabbit hole I would not soon emerge from!</p>
<h3 id="glotio-code-pastebin">glot.io code pastebin</h3>
<p>There are many public pastebins these days, but <a href="https://glot.io/">glot.io</a> distinguishes itself by allowing you to run real code on their server in nearly 50 languages.</p>
<p>It offers both public and private pastes, has an API, and is open source.</p>
<h3 id="firefox-send">Firefox Send</h3>
<p>Firefox Send at <a href="https://send.firefox.com/">send.firefox.com</a> is a browser-based service for securely sharing files temporarily, for only one download during a maximum of 24 hours.</p>
<p>Handy for keeping unwanted bloat out of email, chat, or shared file storage for ephemeral files.</p>
<h3 id="transfersh-command-line-file-sharing">transfer.sh command-line file sharing</h3>
<p>Similarly, <a href="https://transfer.sh/">transfer.sh</a> is a terminal-based file upload and download tool.</p>
<p>As a command-line tool it easily integrates with other standard tools, so you can pipe output from other programs directly to it. If you have sensitive data to share you don’t need to trust the service—you can pipe your data through gpg or some other encryption tool before it leaves your computer.</p>
<p>transfer.sh is open source and can be self-hosted too.</p>
<p>It even has a Tor onion service so uploads and/or downloads can be as private as possible in hostile environments.</p>
<h3 id="doing-what-you-dont-want-to-do">Doing what you don’t want to do</h3>
<p>And finally, some timeless tips for making our human “software” work.</p>
<p>Often just one or two annoying little things can block us from making progress on larger projects that overall we really enjoy. How can you motivate yourself to push ahead when you have work that needs to be done, but you don’t want to do it?</p>
<p>Read the brief but helpful article <a href="https://zenhabits.net/unwanted/">10 Ways to Do What You Don’t Want to Do</a> by Leo Babauta to get some good ideas. A few of the points mentioned especially resonate with me:</p>
<ul>
<li>Why do I need to do it?</li>
<li>What is stopping me?</li>
<li>Embrace that it won’t be fun and do it anyway.</li>
<li>Set constraints.</li>
</ul>
<p>Then do at least a little bit of the work to get started. As our co-worker <a href="/team/mike-heins/">Mike Heins</a> has said to me on a few occasions over the years, you’ll never finish until you start.</p>
A Collaborative Timezone Utilityhttps://www.endpointdev.com/blog/2017/10/a-collaborative-timezone-utility/2017-10-30T00:00:00+00:00Joe Marrero
<div style="float: right; width: 300px; padding: 0 0 1em 1em;">
<div style="padding: 1em; border: 1px solid #ccc; border-radius: 6px;">
<h3 style="margin: 0 0 1rem 0;">Try It Out Yourself</h3>
<p>The code for this project is hosted on <a href="https://github.com/manvscode/timezoner">GitHub and can be cloned from here.</a></p>
</div>
</div>
<p>At <a href="/">End Point Corporation</a>, our team is spread out across 10 time zones. This gives us the advantage of being able to work around the clock on projects. When one co-worker leaves for day, another can take over. Consider this scenario. It’s Monday evening and Martin needs to continue installing software on that Linux cluster, but it’s already 6pm and his wife is going to murder him if he’s not ready to go out for their anniversary dinner. Let’s see who can take over… Ah, yes, Sanjay in Bangalore can continue with the maintenance. Tuesday morning, the client wakes up to be surprised that 16 hours of work was completed in a day. With respect to software development, the same efficiencies can be realized by parallelizing tasks across time-zones. Code reviews and further development can be continued after normal business hours.</p>
<p>With all the blessings of a distributed engineering team, collaborating with co-workers can be, occasionally, challenging. Some of these challenges stem from complexities of our system of time. Every co-worker may be operating in a timezone that is different than yours. Time-zones have an associated offset relative to <a href="https://en.wikipedia.org/wiki/Coordinated_Universal_Time">Coordinated Universal Time (UTC)</a>. These offsets are usually in whole hour increments but they may be any real-valued number.</p>
<p>For example, <a href="https://en.wikipedia.org/wiki/Eastern_Time_Zone">Eastern Standard Time (EST)</a> has an offset of -5 (five hours behind UTC) and <a href="https://en.wikipedia.org/wiki/Indian_Standard_Time">Indian Standard Time (IST)</a> has an offset of 5.5 (five and half hours ahead of UTC). Furthermore, these UTC offsets can be completely arbitrary. In 1995, <a href="https://en.wikipedia.org/wiki/Kiribati">Kiribati</a>, an island nation in the Pacific, changed its UTC offset from -10 to +14 so that all of its outlying islands can share the same time. To further complicate things, some regions may not observe <a href="https://en.wikipedia.org/wiki/Daylight_saving_time">daylight savings time (DST)</a> while other regions do. In fact, in the United States, Indiana started observing DST on April 2, 2006. Some states like Arizona and Hawaii do not observe DST. Other countries, like Australia, have a similar situation where it’s left to local governments to decide whether DST is observed or not. Moreover, although DST usually accounts for adding or subtracting an hour of time, it isn’t always one hour. This has historically changed from time to time.</p>
<p>Now you may begin to imagine the headaches that arise when you need to coordinate with anything involving multiple time-zones. To make all of this easier, you can use a utility that we wrote to do all the time conversions for you. First, you have to add each co-worker’s information to a configuration file stored at ~/.timezoner. This configuration file will describe all of your co-worker’s contact information and their associated IANA time-zone. As an example, this is what the configuration file looks like:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash"><span style="color:#888"># Timezone Email Name OfficePhone MobilePhone</span>
America/New_York <span style="color:#d20;background-color:#fff0f0">"edward@example.com"</span> <span style="color:#d20;background-color:#fff0f0">"Edward Teach "</span> <span style="color:#d20;background-color:#fff0f0">"n/a"</span> <span style="color:#d20;background-color:#fff0f0">"+1 731 555 1234"</span>
America/New_York <span style="color:#d20;background-color:#fff0f0">"henry@dexample.com"</span> <span style="color:#d20;background-color:#fff0f0">"Henry Morgan"</span> <span style="color:#d20;background-color:#fff0f0">"+1 646 555 5678"</span> <span style="color:#d20;background-color:#fff0f0">"+1 954 555 5678"</span>
America/New_York <span style="color:#d20;background-color:#fff0f0">"john@example.com"</span> <span style="color:#d20;background-color:#fff0f0">"John Auger"</span> <span style="color:#d20;background-color:#fff0f0">"n/a"</span> <span style="color:#d20;background-color:#fff0f0">"+1 902 555 1234"</span>
America/Denver <span style="color:#d20;background-color:#fff0f0">"sam@example.com"</span> <span style="color:#d20;background-color:#fff0f0">"Samuel Bellamy"</span> <span style="color:#d20;background-color:#fff0f0">"+1 347 535 1234"</span> <span style="color:#d20;background-color:#fff0f0">"+1 994 555 5678"</span>
America/Los_Angeles <span style="color:#d20;background-color:#fff0f0">"william@example.com"</span> <span style="color:#d20;background-color:#fff0f0">"William Kidd"</span> <span style="color:#d20;background-color:#fff0f0">"+1 330 555 5678"</span> <span style="color:#d20;background-color:#fff0f0">"+1 305 555 1234"</span>
America/Los_Angeles <span style="color:#d20;background-color:#fff0f0">"israel@example.com"</span> <span style="color:#d20;background-color:#fff0f0">"Israel Hands"</span> <span style="color:#d20;background-color:#fff0f0">"+1 507 555 1234"</span> <span style="color:#d20;background-color:#fff0f0">"+1 208 555 5678"</span>
</code></pre></div><p>Now when I need to coordinate a meeting, I can run the utility with the -T option to see each team member’s local time.</p>
<div class="separator" style="clear: both; text-align: center; margin-bottom: 1em;"><a href="/blog/2017/10/a-collaborative-timezone-utility/image-0-big.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="320" data-original-width="1600" height="172" src="/blog/2017/10/a-collaborative-timezone-utility/image-0.png" width="846" /></a></div>
<p>With the -U option, you can display each contact separated in groups based on UTC offset.</p>
<div class="separator" style="clear: both; text-align: center; margin-bottom: 1em;"><a href="/blog/2017/10/a-collaborative-timezone-utility/image-1-big.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="833" data-original-width="1600" height="333" src="/blog/2017/10/a-collaborative-timezone-utility/image-1.png" width="640" /></a></div>
<p>Let us know what you think and if you found this tool helpful.</p>
Converting from Pivotal Tracker to Trello for project managementhttps://www.endpointdev.com/blog/2015/08/converting-from-pivotal-tracker-to/2015-08-03T00:00:00+00:00Josh Lavin
<p>For larger client projects, I find it helpful to maintain a list of tasks, with the ability to re-order, categorize, and mark tasks as completed. Add in the ability to share this list with coworkers or project owners, and you have a recipe for a better record and task-list for development.</p>
<p>I had been using <a href="https://www.pivotaltracker.com/">Pivotal Tracker</a> for this purpose, but I found a lot of its features were too complicated for a small team. On the simpler side, <a href="https://trello.com/">Trello</a> offers project “boards” that meet many needs for project management. Plus, you can accomplish a lot with the free level of Trello.</p>
<h3 id="no-import">No import</h3>
<p>After being convinced that switching from Pivotal to Trello was the right move for my current project, I was dismayed to find that Trello offers no Import functionality, at least that I could find for front-end users. (They do have an API, but I didn’t relish taking time to learn this.) I could easily export my Pivotal project, but how to get those tasks into Trello cards?</p>
<h3 id="one-idea">One idea</h3>
<p>In my search of Trello for an import feature, I found a feature called <em>Email-to-board</em>. Trello provides a custom email address for each Board you create, allowing you to send emails to this address, containing information for a new Trello card. (This email address is unique for each board, containing random letters and numbers, so only the Board owner can use it.)</p>
<p>What if I wrote a quick script that processed a Pivotal CSV export file, and sent an email to Trello for each row (task) in the file? The script might send out quite a few emails, but would it work? Time to try it.</p>
<h3 id="perl-to-the-rescue">Perl to the rescue</h3>
<p>I started cooking up a simple <a href="https://www.perl.org/">Perl</a> script to test the idea. With the help of some <a href="https://metacpan.org/">CPAN</a> modules to easily process the CSV file and send the emails, I landed on something that worked. After running the script, each row in the CSV export became an email to my Trello board, containing the item’s title, description, estimate (difficulty level), and list of tasks required to complete it.</p>
<p>The script should work for most exports from Pivotal Tracker, and I have published it to my <a href="https://github.com/jdigory">GitHub account</a>, in case it is helpful for others who decide to move to Trello.</p>
<p><strong><a href="https://github.com/jdigory/pivotal2trello">Find the pivotal2trello script on GitHub.</a></strong></p>
<p>If you happen to try it, let me know if you experience any problems, or have any suggestions!</p>
IPython Tips and Trickshttps://www.endpointdev.com/blog/2015/06/ipython-tips-and-tricks/2015-06-18T00:00:00+00:00Kannan Ponnusamy
<p>Recently I have been working on Python automation scripts. Very often I use IPython to develop/debug the code.</p>
<p>IPython is an advanced interactive python shell. It is a powerful tool which has many more features. However, here I would like to share some of the cool tricks of IPython.</p>
<h3 id="getting-help">Getting help</h3>
<p>Typing object_name? will print all sorts of details about any object, including docstrings, function definition lines (for call arguments) and constructor details for classes.</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-python" data-lang="python">In [<span style="color:#00d;font-weight:bold">1</span>]: <span style="color:#080;font-weight:bold">import</span> <span style="color:#b06;font-weight:bold">datetime</span>
In [<span style="color:#00d;font-weight:bold">2</span>]: datetime.datetime<span style="color:#a61717;background-color:#e3d2d2">?</span>
Docstring:
datetime(year, month, day[, hour[, minute[, second[, microsecond[,tzinfo]]]]])
The year, month <span style="color:#080">and</span> day arguments are required. tzinfo may be <span style="color:#080;font-weight:bold">None</span>, <span style="color:#080">or</span> an
instance of a tzinfo subclass. The remaining arguments may be ints <span style="color:#080">or</span> longs.
File: /System/Library/Frameworks/Python.framework/Versions/<span style="color:#00d;font-weight:bold">2.7</span>/lib/python2<span style="color:#00d;font-weight:bold">.7</span>/lib-dynload/datetime.so
Type: <span style="color:#038">type</span>
</code></pre></div><h3 id="magic-commands">Magic commands</h3>
<h3 id="edit">Edit</h3>
<p>This will bring up an editor to type multiline code and execute the resulting code.</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-python" data-lang="python">In [<span style="color:#00d;font-weight:bold">3</span>]: %edit
IPython will make a temporary file named: /var/folders/xh/<span style="color:#00d;font-weight:bold">2</span>m0ydjs51qxd_3y2k7x50hjc0000gn/T/ipython_edit_jnVJ51/ipython_edit_NdnenL.py
</code></pre></div><div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-python" data-lang="python">In [<span style="color:#00d;font-weight:bold">3</span>]: %edit -p
</code></pre></div><p>This will bring up the editor with the same data as the previous time it was used or saved. (in the current session)</p>
<h3 id="run-a-script">Run a script</h3>
<p>This will execute the script and print the results.</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-python" data-lang="python">In [<span style="color:#00d;font-weight:bold">12</span>]: %run date_sample.py
Current date <span style="color:#080">and</span> time: <span style="color:#00d;font-weight:bold">2015</span>-<span style="color:#00d;font-weight:bold">06</span>-<span style="color:#00d;font-weight:bold">18</span> <span style="color:#00d;font-weight:bold">16</span>:<span style="color:#00d;font-weight:bold">10</span>:<span style="color:#00d;font-weight:bold">34.444674</span>
Or like this: <span style="color:#00d;font-weight:bold">15</span>-<span style="color:#00d;font-weight:bold">06</span>-<span style="color:#00d;font-weight:bold">18</span>-<span style="color:#00d;font-weight:bold">16</span>-<span style="color:#00d;font-weight:bold">10</span>
Week number of the year: <span style="color:#00d;font-weight:bold">24</span>
Weekday of the week: <span style="color:#00d;font-weight:bold">4</span>
</code></pre></div><h3 id="debug">Debug</h3>
<p>Activate the interactive debugger.</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-python" data-lang="python">In [<span style="color:#00d;font-weight:bold">15</span>]: %run date.py
Current date <span style="color:#080">and</span> time: <span style="color:#00d;font-weight:bold">2015</span>-<span style="color:#00d;font-weight:bold">06</span>-<span style="color:#00d;font-weight:bold">18</span> <span style="color:#00d;font-weight:bold">16</span>:<span style="color:#00d;font-weight:bold">12</span>:<span style="color:#00d;font-weight:bold">32.417691</span>
Or like this: ---------------------------------------------------------------------------
<span style="color:#b06;font-weight:bold">TypeError</span> Traceback (most recent call last)
/Users/kannan/playground/date.py <span style="color:#080">in</span> <module>()
<span style="color:#00d;font-weight:bold">3</span>
<span style="color:#00d;font-weight:bold">4</span> <span style="color:#038">print</span> <span style="color:#d20;background-color:#fff0f0">"Current date and time: "</span> , datetime.datetime.now()
----> <span style="color:#00d;font-weight:bold">5</span> <span style="color:#038">print</span> <span style="color:#d20;background-color:#fff0f0">"Or like this: "</span> ,datetime.datetime.strftime(<span style="color:#d20;background-color:#fff0f0">"%y-%m-</span><span style="color:#33b;background-color:#fff0f0">%d</span><span style="color:#d20;background-color:#fff0f0">-%H-%M"</span>)
<span style="color:#00d;font-weight:bold">6</span> <span style="color:#038">print</span> <span style="color:#d20;background-color:#fff0f0">"Week number of the year: "</span>, datetime.date.today().strftime(<span style="color:#d20;background-color:#fff0f0">"%W"</span>)
<span style="color:#00d;font-weight:bold">7</span> <span style="color:#038">print</span> <span style="color:#d20;background-color:#fff0f0">"Weekday of the week: "</span>, datetime.date.today().strftime(<span style="color:#d20;background-color:#fff0f0">"%w"</span>)
<span style="color:#b06;font-weight:bold">TypeError</span>: descriptor <span style="color:#d20;background-color:#fff0f0">'strftime'</span> requires a <span style="color:#d20;background-color:#fff0f0">'datetime.date'</span> <span style="color:#038">object</span> but received a <span style="color:#d20;background-color:#fff0f0">'str'</span>
In [<span style="color:#00d;font-weight:bold">16</span>]: %debug
> /Users/kannan/playground/date.py(<span style="color:#00d;font-weight:bold">5</span>)<module>()
<span style="color:#00d;font-weight:bold">4</span> <span style="color:#038">print</span> <span style="color:#d20;background-color:#fff0f0">"Current date and time: "</span> , datetime.datetime.now()
----> <span style="color:#00d;font-weight:bold">5</span> <span style="color:#038">print</span> <span style="color:#d20;background-color:#fff0f0">"Or like this: "</span> ,datetime.datetime.strftime(<span style="color:#d20;background-color:#fff0f0">"%y-%m-</span><span style="color:#33b;background-color:#fff0f0">%d</span><span style="color:#d20;background-color:#fff0f0">-%H-%M"</span>)
<span style="color:#00d;font-weight:bold">6</span> <span style="color:#038">print</span> <span style="color:#d20;background-color:#fff0f0">"Week number of the year: "</span>, datetime.date.today().strftime(<span style="color:#d20;background-color:#fff0f0">"%W"</span>)
ipdb>
</module></module>
</code></pre></div><p>I made a error in the line number 5, it should have to look like this. So %debug command took me into the Python debugger.</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-python" data-lang="python"><span style="color:#038">print</span> <span style="color:#d20;background-color:#fff0f0">"Or like this: "</span> ,datetime.datetime.now().strftime(<span style="color:#d20;background-color:#fff0f0">"%y-%m-</span><span style="color:#33b;background-color:#fff0f0">%d</span><span style="color:#d20;background-color:#fff0f0">-%H-%M"</span>)
</code></pre></div><h3 id="save">Save</h3>
<p>This will save the specified lines to a given file. You can pass any number of arguments separated by space.</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-python" data-lang="python">In [<span style="color:#00d;font-weight:bold">21</span>]: %save hello.py <span style="color:#00d;font-weight:bold">1</span>-<span style="color:#00d;font-weight:bold">2</span> <span style="color:#00d;font-weight:bold">2</span>-<span style="color:#00d;font-weight:bold">3</span>
The following commands were written to file <span style="color:#a61717;background-color:#e3d2d2">`</span>hello.py<span style="color:#a61717;background-color:#e3d2d2">`</span>:
<span style="color:#080;font-weight:bold">import</span> <span style="color:#b06;font-weight:bold">datetime</span>
datetime.datetime<span style="color:#a61717;background-color:#e3d2d2">?</span>
%edit
%edit -p
</code></pre></div><h3 id="recall">Recall</h3>
<p>Repeat a command, or get command to input line for editing.</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-python" data-lang="python">In [<span style="color:#00d;font-weight:bold">28</span>]: %recall <span style="color:#00d;font-weight:bold">21</span>
In [<span style="color:#00d;font-weight:bold">29</span>]: <span style="color:#080;font-weight:bold">import</span> <span style="color:#b06;font-weight:bold">datetime</span>
</code></pre></div><h3 id="timeit">Timeit</h3>
<p>Time execution of a Python statement or expression</p>
<p>It can be one line or multiline statement. In a one liner we can pass through multiple ones separated by semicolon.</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-python" data-lang="python">In [<span style="color:#00d;font-weight:bold">33</span>]: %timeit <span style="color:#038">range</span>(<span style="color:#00d;font-weight:bold">100</span>)
<span style="color:#00d;font-weight:bold">1000000</span> loops, best of <span style="color:#00d;font-weight:bold">3</span>: <span style="color:#00d;font-weight:bold">752</span> ns per loop
</code></pre></div><h3 id="shell-commands">Shell Commands</h3>
<p>Basic UNIX shell integration (you can run simple shell commands such as cp, ls, rm, cp, etc. directly from the ipython command line)</p>
<p>To execute any other shell commands we just need to add “!” beginning of the command line. We can assign the result of the system command to a Python variable to further use.</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-python" data-lang="python">In [<span style="color:#00d;font-weight:bold">38</span>]: list_of_files = <span style="color:#a61717;background-color:#e3d2d2">!</span>ls
In [<span style="color:#00d;font-weight:bold">39</span>]: list_of_files
Out[<span style="color:#00d;font-weight:bold">39</span>]:
[<span style="color:#d20;background-color:#fff0f0">'lg-live-build'</span>,
<span style="color:#d20;background-color:#fff0f0">'lg-live-image'</span>,
<span style="color:#d20;background-color:#fff0f0">'lg-peruse-a-rue'</span>,
<span style="color:#d20;background-color:#fff0f0">'lg_chef'</span>,
<span style="color:#d20;background-color:#fff0f0">'lg_cms'</span>,
<span style="color:#d20;background-color:#fff0f0">'playground'</span>]
</code></pre></div><h3 id="history">History</h3>
<p>Print input history, with most recent last.</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-python" data-lang="python">In [<span style="color:#00d;font-weight:bold">41</span>]: %history <span style="color:#00d;font-weight:bold">20</span>-<span style="color:#00d;font-weight:bold">22</span>
ls
<span style="color:#080;font-weight:bold">import</span> <span style="color:#b06;font-weight:bold">datetime</span>
datetime.datetime.now()
</code></pre></div><div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-python" data-lang="python">%history ~<span style="color:#00d;font-weight:bold">1</span>/<span style="color:#00d;font-weight:bold">4</span> <span style="color:#888">#Line 4, from last session</span>
</code></pre></div><p>This will list the previous session history.</p>
<h3 id="pastebin">Pastebin</h3>
<p>This will upload the specifed input commands to Github’s Gist paste bin, and display the URL</p>
<p>It will upload the code as anonymous user</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-python" data-lang="python">In [<span style="color:#00d;font-weight:bold">43</span>]: %pastebin [-d <span style="color:#a61717;background-color:#e3d2d2">“</span>Date Example<span style="color:#a61717;background-color:#e3d2d2">”</span>] <span style="color:#00d;font-weight:bold">20</span>-<span style="color:#00d;font-weight:bold">23</span>
Out[<span style="color:#00d;font-weight:bold">43</span>]: <span style="color:#d20;background-color:#fff0f0">u</span><span style="color:#d20;background-color:#fff0f0">'https://gist.github.com/a660948b8323280a0d27'</span>
</code></pre></div><p>For more info on this topic:</p>
<p><a href="http://ipython.org/ipython-doc/dev/interactive/tutorial.html">http://ipython.org/ipython-doc/dev/interactive/tutorial.html</a></p>
<p><a href="http://ipython.org/ipython-doc/dev/interactive/magics.html">http://ipython.org/ipython-doc/dev/interactive/magics.html</a></p>
On End Point’s Development Environmenthttps://www.endpointdev.com/blog/2015/03/on-end-points-development-environment/2015-03-11T00:00:00+00:00Steph Skardal
<p>A few recent conversations have sparked my interest in writing up a blog post that summarizes the familiar elements of our development environments. The majority of End Pointers work remotely, but many of the tools listed below are common to many developers.</p>
<ul>
<li><strong>ssh/sftp</strong>: We do primarily remote development. Many of us are familiar with local development, but remote development with camps (see next point) is typically the most efficient arrangement in working with multiple development instances that are accessible to clients for testing and staging.</li>
<li><strong>camps</strong>: <a href="http://www.devcamps.org/">DevCamps</a> are a tool specific to and created by End Point, which are development instances with an entire webserver, database, and app server stack, similar to containers like <a href="https://www.docker.com/">Docker</a>. Check out <a href="http://www.devcamps.org/">the DevCamps website</a> for more information.</li>
<li><strong>vim/emacs/nano</strong>: While most of our employees use vim or emacs for command-line editors, nano is an inefficient but easy to use editor that we can suggest to new developers. Not many of us use <a href="https://en.wikipedia.org/wiki/Integrated_development_environment">IDEs</a>, if at all.</li>
<li><strong>screen/tmux</strong>: screen and tmux are our preferred terminal multitasking and sharing.</li>
<li><strong>command-line database interaction (specifically psql and mysql ad-hoc querying)</strong>: Working with an SQL database through an <a href="https://en.wikipedia.org/wiki/Object-relational_mapping">ORM</a> like <a href="http://guides.rubyonrails.org/active_record_basics.html">Active Record</a>, <a href="http://www.dbix-class.org/">DBIC</a>, etc. is not enough for us.</li>
<li><strong>*nix / basic command-line interaction</strong>: This topic could make up its own blog post, but some of the tools we use frequently are netstat/ss, ifconfig/ip, lsof, ps/top/htop/atop, free, df, nice/ionice, tail -f, sort, uniq -c, grep.</li>
<li><strong>git & <a href="https://github.com/">github</a></strong>: Not uncommon to devshops these days, git is the most popular version control system, and github an extremely popular host of both open source and private repositories.</li>
<li><strong>IRC, Skype, Google Hangouts, <a href="https://appear.in/">appear.in</a>, <a href="https://talky.io/">talky.io</a>, <a href="https://glideroom.com/">glideroom.com</a>, Google Voice, & <em>gasp</em> regular phones</strong>: As remote developers, we communicate often and there are a number of tools available in the communication space that we leverage.</li>
</ul>
<p>It’s interesting to note that if any new developers come in with a preference for a trendy new tool, while we are happy to let them work in an environment that allows them to be efficient, ultimately we can’t provide support for those tools that we are unfamiliar with.</p>
Easier IE Site Testing With RemoteIEhttps://www.endpointdev.com/blog/2014/12/easier-ie-site-testing-with-remoteie/2014-12-02T00:00:00+00:00Greg Davidson
<p>Microsoft recently announced a new service which I’m finding very useful. <a href="https://remote.modern.ie/">RemoteIE</a> lets you test your sites with IE (currently version 11) on Windows 10 Technical Preview. The service runs in Azure RemoteApp which is available for <a href="https://web.archive.org/web/20171201025431/https://docs.microsoft.com/en-us/windows-server/remote/remote-desktop-services/clients/remote-desktop-clients">several clients</a> including Android, iOS and Windows Phone. What’s great about this is that you do not have to install and maintain your own virtual machine with VirtualBox or VMWare.</p>
<p><a href="https://remote.modern.ie/" title="More information about RemoteIE from Microsoft"><img alt="RemoteIE" border="0" height="484" src="/blog/2014/12/easier-ie-site-testing-with-remoteie/image-0.png" title="remoteIE.png" width="615"/></a></p>
<p>To use RemoteIE you’ll need a valid Microsoft account—it’s easy to sign up if you don’t already one. Once you have an account and have downloaded & installed the Azure RemoteApp client of your choice it’s just a matter of starting it up and logging in. Happy Testing!</p>
Looking at development environments with DevCamps and Vagranthttps://www.endpointdev.com/blog/2014/08/looking-at-development-environments/2014-08-25T00:00:00+00:00Spencer Christensen
<p>For most web developers, you have practices and tools that you are used to using to do your work. And for most web developers this means setting up your workstation with all the things you need to do your editing, compiling, testing, and pushing code to some place for sharing or deployment. This is a very common practice even though it is fraught with problems- like getting a database setup properly, configuring a web server, any other services (memcached, redis, mongodb, etc), and many more issues.</p>
<p>Hopefully at some point you realize the pain that is involved in doing everything on your workstation directly and start looking for a better way to do web development. In this post I will be looking at some ways to do this better: using a virtual machine (VM), Vagrant, and DevCamps.</p>
<h3 id="using-a-vm-for-development">Using a VM for development</h3>
<p>One way to improve things is to use a local virtual machine for your development (for example, using VirtualBox, or VMware Fusion). You can edit your code normally on your workstation, but then execute and test it in the VM. This also makes your workstation “clean”, moving all those dependencies (like a database, web server, etc.) off your workstation and into the VM. It also gets your dev environment closer to production, if not identical. Sounds nice, but let’s break down the pros and cons.</p>
<h4 id="benefits-of-using-a-vm">Benefits of using a VM</h4>
<ul>
<li>Dev environment closely matches production.</li>
<li>Execute and test code in a dedicated machine (not your workstation directly).</li>
<li>Allows for multiple projects to be worked on concurrently (one VM per project).</li>
<li>Exposes the developer to the Operations (systems administration) side of the web application (always a good thing).</li>
<li>Developer can edit files using their favorite text editor locally on the workstation (but will need to copy files to the VM as needed).</li>
</ul>
<h4 id="problems-with-using-a-vm">Problems with using a VM</h4>
<ul>
<li>Need to create and configure the VM. This could be very time consuming and error prone.</li>
<li>Still need to install and configure all services and packages. This could also be time consuming and error prone.</li>
<li>Backups of your work/configuration/everything are your own responsibility (extremely unlikely to happen).</li>
<li>Access to your dev environment is extremely limited, thus probably only you can access it and test things on it. No way for a QA engineer or business owner to test/demo your work.</li>
<li>Inexperienced developers can break things, or change them to no longer match production (install arbitrary packages, different versions than what is in production, screw up the db, screw up Apache configuration, etc.).</li>
<li>If working with an established database, then downloading a dump, installing, and getting the database usable is time consuming and error prone. (“I just broke my dev database!” can be a complete blocker for development.)</li>
<li>The developer needs to set up networking for the VM in order to ssh to it, copy files back and forth, and point a web browser to it. This may include manually setting up DNS, or /etc/hosts entries, or port forwarding, or more complex setups.</li>
<li>If using SSL with the web application, then the developer also needs to generate and install the SSL cert and configure the web server correctly.</li>
</ul>
<h3 id="vagrant">Vagrant</h3>
<p>What is <a href="https://www.vagrantup.com">Vagrant</a>? It is a set of tools to make it easier to use a virtual machine for your web development. It attempts to lessen many of the problems listed above through the use of automation. By design it also makes some assumptions about how you are using the VM. For example, it assumes that you have the source code for you project in a directory somewhere directly on your workstation and would prefer to use your favorite text editor on those files. Instead of expecting you to continually push updated files to your VM, it sets up a corresponding directory on the VM and keeps the two in sync for you (using either shared folders, NFS, Samba, or rsync). It also sets up the networking for accessing the VM, usually with port forwarding, so you don’t have to worry about that.</p>
<h4 id="benefits-of-vagrant">Benefits of Vagrant</h4>
<ul>
<li><em><strong>Same as those listed above for using a VM, plus…</strong></em></li>
<li>Flexible configuration (Vagrantfile) for creating and configuring the VM.</li>
<li>Automated networking for the VM with port forwarding. Abstracted ssh access (don’t need to set up a hostname for the VM, simply type <code>vagrant ssh</code> to connect). Port forwarded browser access to the VM (usually http://localhost:8080, but configurable).</li>
<li>Synced directory between your workstation and the VM for source code. Allows for developers to use their favorite text editor locally on their workstation without needing to manually copy files to the VM.</li>
<li>Expects the use of a configuration management system (like puppet, chef, salt, or bash scripts) to “provision” the VM (which could help with proper and consistent setup).</li>
<li>Through the use of <a href="https://vagrantcloud.com/">Vagrant Cloud</a> you can get a generated url for others to access your VM (makes it publicly available through a tunnel created with the command <code>vagrant share</code>).</li>
<li>Configuration (Vagrantfile and puppet/chef/salt/etc.) files can be maintained/reviewed by Operations engineers for consistency with production.</li>
</ul>
<h4 id="problems-with-vagrant">Problems with Vagrant</h4>
<ul>
<li>Still need to install and configure all services and packages. This is lessened with the use of a configuration management tool like puppet, but you still need to create/debug/maintain the puppet configuration and setup.</li>
<li>Backups of your work/configuration/everything are your own responsibility (extremely unlikely to happen). This may be lessened for VM configuration files, assuming they are included in your project’s VCS repo along with your source code.</li>
<li>Inexperienced developers can still break things, or change them to no longer match production (install arbitrary packages, different versions than what is in production, screw up the db, screw up Apache configuration, etc.).</li>
<li>If working with an established database, then downloading a dump, installing, and getting the database usable is time consuming and error prone. (“I just broke my dev database!” can be a complete blocker for development.)</li>
<li>If using SSL with the web application, then the developer also needs to generate and install the SSL cert and configure the web server correctly. This might be lessened if puppet (or whatever) is configured to manage this for you (but then you need to configure puppet to do that).</li>
</ul>
<h3 id="devcamps">DevCamps</h3>
<p>The <a href="http://devcamps.org">DevCamps</a> system takes a different approach. Instead of using VMs for development, it utilizes a shared server for all development. Each developer has their own account on the camps server and can create/update/delete “camps” (which are self-contained environments with all the parts needed). There is an initial setup for using camps which needs thorough understanding of the web application and all of its dependencies (OS, packages, services, etc.). For each camp, the system will create a directory for the user with everything related to that camp in it, including the web application source code, their own web server configuration, their own database with its own configuration, and any other resources. Each camp is assigned a camp number, and all services for that camp run on different ports (based on the camp number). For example, camp 12 may have Apache running on ports 9012 (HTTP) and 9112 (HTTPS) and MySQL running on port 8912. The developer doesn’t need to know these ports, as tools allow for easier access to the needed services (commands like <code>mkcamp</code>, <code>re</code> for restarting services, <code>mysql_camp</code> for access to the database, etc.).</p>
<p>DevCamps has been designed to address some of the pain usually associated with development environments. Developers usually do not need to install anything, since all dependencies should already be installed on the camps server (which should be maintained by an Operations engineer who can keep the packages, versions, etc. consistent with production). Having all development on a server allows Operations engineers to backup all dev work fairly easily. Databases do not need to be downloaded, manually setup, or anything- they should be set up initially with the camps system and then running <code>mkcamp</code> clones the database and sets it up for you. Running <code>refresh-camp --db</code> allows a developer to delete their camp’s database and get a fresh clone, ready to use.</p>
<h4 id="benefits-of-devcamps">Benefits of DevCamps</h4>
<ul>
<li>Each developer can create/delete camps as needed, allowing for multiple camps at once and multiple projects at once.</li>
<li>Operations engineers can manage/maintain all dependencies for development, ensuring everything is consistent with production.</li>
<li>Backups of all dev work is easy (Operations engineer just needs to backup the camps server).</li>
<li>Developer does not need to configure services (camp templating system auto-generates needed configuration for proper port numbers), such as Apache, nginx, unicorn, MySQL, Postgres, etc.</li>
<li>SSL certificates can be easily shared/generated/installed/etc. automatically with the <code>mkcamp</code> script. Dev environments can easily have HTTPS without the developer doing anything.</li>
<li>Developers should not have permissions to install/change system packages or services. Thus inexperienced developers should not be able to break the server, other developer’s environments, install arbitrary software. Screwing up their database or web server config can be fixed by either creating a new camp, refreshing their existing one, or an Operations engineer can easily fix it for them (since it is on a central server they would already have access to, and not need to worry about how to access some VM who knows where).</li>
</ul>
<h4 id="problems-with-devcamps">Problems with DevCamps</h4>
<ul>
<li>Since all camps live on a shared server running on different ports, this will not closely match production in that way. However, this may not be significant if nearly everything else does closely match production.</li>
<li>Adding a new dependency (for example, adding mongodb, or upgrading the version of Apache) may require quite a bit of effort and will affect all camps on the server- Operations engineer will need to install the needed packages and add/change the needed configuration to the camps system and templates.</li>
<li>Using your favorite text editor locally on your workstation doesn’t really work since all code lives on the server. It is possible to SFTP files back and forth, but this can be tedious and error prone.</li>
<li>Many aspects of the Operations (systems administration) side of the web application are hidden from the developer (this might also be considered a benefit).</li>
<li>All development is on a single server, which may be a single point of failure (if the camps server is down, then all development is blocked for all developers).</li>
<li>One camp can use up more CPU/RAM/disk/etc. then others and affect the server’s load, affecting the performance of all other camps.</li>
</ul>
<h3 id="concluding-thoughts">Concluding Thoughts</h3>
<p>It seems that Vagrant and DevCamps certainly have some good things going for them. I think it might be worth some thought and effort to try to meld the two together somehow, to take the benefits of both and reduce the problems as much as possible. Such a system might look like this:</p>
<ul>
<li>Utilize vagrant commands and configuration, but have all VMs live on a central VM server. Thus allowing for central backups and access.</li>
<li>Source code and configuration lives on the server/VM but a synced directory is set up (sshfs mount point?) to allow for local editing of text files on the workstation.</li>
<li>VMs created should have restricted access, preventing developers from installing arbitrary packages, versions, screwing up the db, etc.</li>
<li>Configuration for services (database, web server, etc.) should be generated/managed by Operations engineers for consistency (utilizing puppet/chef/salt/etc.).</li>
<li>Databases should be cloned from a local copy on the VM server, thus avoiding the need to download anything and reducing setup effort.</li>
<li>SSL certs should be copied/generated locally on the VM server and installed as appropriate.</li>
<li>Sharing access to a VM should not depend on Vagrant Cloud, but instead should use some sort of internal service on the VM server to automate VM hostname/DNS for browser and ssh access to the VM.</li>
</ul>
<p>I’m sure there are more pros and cons that I’ve missed. Add your thoughts to the comments below. Thanks.</p>
Android Developer Tools via Google Chromehttps://www.endpointdev.com/blog/2014/06/android-developer-tools-via-google/2014-06-11T00:00:00+00:00Zed Jensen
<p>Recently I was working on a website on my Android phone, and I found myself needing Chrome’s Developer Tools. However, Developer Tools are not included in the Android version of Chrome for many reasons, including lack of screen real estate.</p>
<p>So, I looked around, and I found a solution: using a USB cable and ADB (<a href="https://developer.android.com/tools/help/adb.html">Android Debug Bridge</a>), you can do debugging on an Android device with Chrome’s Developer Tools <em>from your desktop.</em></p>
<p>To show you exactly what I mean, here’s a short video demonstrating this:</p>
<div class="separator" style="clear: both; text-align: center;">
<iframe width="560" height="315" src="https://www.youtube-nocookie.com/embed/ut7NWQZVXEk?rel=0" frameborder="0" allowfullscreen></iframe>
</div>
<p>So, how does one work this magic? There are several ways, but I’ll talk about the one that I used. For this method, you need to have Google Chrome version 31 or higher installed on both your Android device and your development machine.</p>
<p>First, you have to enable Android debugging on your device. From <a href="https://developer.android.com/tools/device.html">android.com</a>:</p>
<blockquote>
<ul>
<li>On most devices running Android 3.2 or older, you can find the option under <strong>Settings > Applications > Development</strong>.</li>
<li>On Android 4.0 and newer, it’s in <strong>Settings > Developer options</strong>.
<ul>
<li><strong>Note:</strong> On Android 4.2 and newer, <strong>Developer options</strong> is hidden by default. To make it available, go to <strong>Settings > About phone</strong> and tap <strong>Build number</strong> seven times. Return to the previous screen to find <strong>Developer options</strong>.</li>
</ul>
</li>
</ul>
</blockquote>
<p>Next, connect your device with a USB cable and, on your development machine, go to <strong>about:inspect</strong> and check <strong>Discover USB Devices</strong>. After a second or two, your device should show up like this:</p>
<p><a href="/blog/2014/06/android-developer-tools-via-google/image-0-big.jpeg" imageanchor="1"><img border="0" src="/blog/2014/06/android-developer-tools-via-google/image-0.jpeg"/></a></p>
<p>To open Dev Tools for a tab, just click “inspect” below it. The buttons next to “inspect” only appear for tabs open in Chrome, but you can open Developer Tools for any app that uses WebView, whether it’s in Chrome or not.</p>
<p>And there you go! Fully featured Chrome Developer Tools for your Android device on your development machine. More information, including a way to do this with earlier versions of Chrome, can be found at the <a href="https://developer.chrome.com/devtools/docs/remote-debugging">Android Developer site</a>.</p>
ZNC: An IRC Bouncerhttps://www.endpointdev.com/blog/2014/03/znc-irc-bouncer/2014-03-27T00:00:00+00:00Greg Davidson
<h3 id="kickin-it-old-skool">Kickin’ it Old Skool</h3>
<p>At End Point, we use IRC extensively for group chat and messaging. Prior to starting here I had been an occasional IRC user—asking questions about various open source projects on <a href="http://freenode.net/">Freenode</a> and helping others as well. When I began to use IRC daily, I ran into a few things that bugged me and thought I would write about what I have done to mitigate those. While it might not be as fancy as <a href="https://campfirenow.com/">Campfire</a>, <a href="https://www.hipchat.com/">HipChat</a> or <a href="https://slack.com/">Slack</a> I’m happy with my setup now.</p>
<h3 id="what-did-i-miss">What did I miss?</h3>
<p>The first thing that annoyed me about IRC was the lack of <em>persistence</em>. If I wasn’t logged on to the server, I missed out on the action. This was exacerbated by the fact that I live in the Pacific Time zone and lots of discussion takes place before my work day begins. Some people on our team solve this issue by running a terminal-based IRC client (like <a href="https://irssi.org/">irssi</a> or <a href="https://weechat.org/">WeeChat</a> inside a tmux or GNU Screen session on a remote server. This approach works well until the server needs to be rebooted (e.g. for OS or kernel updates etc.). It also introduces the limitation of using a terminal-based client which isn’t for everyone.</p>
<h3 id="test-driving-irc-clients">Test Driving IRC Clients</h3>
<p>The next challenge was finding a good IRC client. In my attempt to test-drive several clients side-by-side I quickly discovered that each client with a direct connection to the server requires its own unique nick (e.g. greg, greg_, greg_2 etc.). I also wanted to be able to use an IRC client on my phone without having to use multiple nicks. At this point I did a little research and determined that a bouncer might be helpful.</p>
<h3 id="enter-the-bouncer">Enter the Bouncer</h3>
<p>A bouncer connects to an IRC server on your behalf and maintains a persistent connection. Rather than connecting directly to IRC server you connect to the bouncer. This allows you to remain connected to IRC while you are offline.</p>
<img alt="Bouncers" border="0" height="389" src="/blog/2014/03/znc-irc-bouncer/image-0.png" title="bouncers.png" width="609"/>
<p><a href="https://wiki.znc.in/ZNC">ZNC</a> is the bouncer software I have been using and it has been great. Using ZNC allows you to connect multiple clients at the same time. E.g. a phone and laptop using a single nick. ZNC also maintains a buffer of the most recent conversations while your clients are not connected. The buffer can be configured to suit your needs (per-channel, per-server etc.). In my case it’s buffering the past 100 messages and when I connect my IRC client they are automatically played back and the buffer is cleared. This has solved my issue with missing discussions and context when my IRC client isn’t running.</p>
<h3 id="irc-to-go">IRC to go</h3>
<p>Another benefit of this setup has been the ability to connect to IRC on my phone (with the same nick as my laptop). I have used Mobile Colloquy and Palaver on iOS and both work quite well. There are also ZNC modules on GitHub that enable ZNC to send push notifications for mentions and private messages. Mobile Colloquy does a good job of this—I have not yet tried this with Palaver yet.</p>
<h3 id="modules">Modules</h3>
<p>ZNC comes with a number of built in modules and allows users to develop their own as well (Perl, C++, Python or Tcl). The following are some of the ones I use:</p>
<ul>
<li>chansaver: keeps you connected to all of the channels you’ve joined</li>
<li>simple_away: update your away message when all of your clients have disconnected / detached</li>
<li>autoreply</li>
<li>log: logs all discussions</li>
<li>web admin: web interface for viewing and editing your ZNC config</li>
</ul>
<h3 id="running-znc">Running ZNC</h3>
<p>ZNC is easy to compile and install and there are also packages available for some Linux distros. After testing it out locally for a while I installed it on a small VPS and that has been working well. <a href="https://digitalocean.com/">Digital Ocean</a> has published an article on <a href="https://www.digitalocean.com/community/articles/how-to-install-znc-an-irc-bouncer-on-an-ubuntu-vps">how to install ZNC on a Ubuntu VPS</a> if you are interested in learning more about it. Also, the <a href="http://wiki.znc.in/ZNC">ZNC wiki</a> has lots of helpful information on how to install and configure ZNC.</p>
Provisioning a Development Environment with Packer, Part 2https://www.endpointdev.com/blog/2014/03/provisioning-development-environment_14/2014-03-14T00:00:00+00:00Mike Farmer
<p>In my <a href="/blog/2014/03/provisioning-development-environment/">previous post</a> on provisioning a development environment with <a href="https://www.packer.io">Packer</a> I walked through getting a server setup with an operating system installed. This post will be focused setting up <a href="https://www.ansible.com">Ansible</a> so that I can setup my development environment just the way I like it. Packer supports many different methods for provisioning. After playing with some of them, I decided that Ansible was a good mix of simplicity and functionality.</p>
<p>A Packer provisioner is simply a configuration template that is added to the json configuration file. The “provisioners” section of the configuration file takes an array of json objects which means that you aren’t stuck with just one kind of provisioner. For example, you could run some shell scripts using the <a href="https://www.packer.io/docs/provisioners/shell.html">shell provisioner</a>, then upload some files using the <a href="https://www.packer.io/docs/provisioners/file.html">File Uploads</a> provisioner, followed by your devops tool of choice (<a href="https://web.archive.org/web/20150706082720/https://www.packer.io/docs/provisioners/puppet-masterless.html">puppet</a>, <a href="https://web.archive.org/web/20190508114914/http://packer.io/docs/provisioners/salt-masterless.html">salt</a>, <a href="https://web.archive.org/web/20210615181646/https://www.packer.io/docs/provisioners/chef-solo">chef</a>, or <a href="https://web.archive.org/web/20210615182101/https://www.packer.io/docs/provisioners/ansible-local">ansible</a>). You can even <a href="https://web.archive.org/web/20160405051550/https://www.packer.io/docs/extend/provisioner.html">roll-your-own</a> provisioner if desired. Here’s an example provisioner setup for the shell provisioner:</p>
<pre tabindex="0"><code>{
"variables": {...},
"builders" : [...],
"provisioners" [
{
"type": "shell",
"inline": [ "echo foo" ]
}
]
}
</code></pre><h3 id="sudo-and-user-considerations">Sudo and User Considerations</h3>
<p>Packer will login to the server over ssh and run your provisioners. The big headache that always comes out of this is some provisioners require sudo, or being logged in as root, to run their commands. Packer, however, will login as the user that was created during the build stage (see the “builders” section in the code snippet in the <a href="/blog/2014/03/provisioning-development-environment/">previous post</a>). There are a couple of ways to handle this. First, you can do everything in packer as root. I don’t love this approach because I like to simulate the way that I setup a machine by hand and I never login as root if I can help it. The second method is to grant your user sudo access. This gets a little tricky so I’ll just show the a code snippet and then explain it below.</p>
<pre tabindex="0"><code>{
"type": "shell",
"execute_command": "echo '{{user `ssh_pass`}}' | {{ .Vars }} sudo -E -S sh '{{ .Path }}'",
"inline": [
"echo '%sudo ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers"
]
}
</code></pre><p>Utilizing the shell provisioner, the execute_command option is used to specify to the provisioner that whenever a command is run, use this command. The commands provided to the inline array are compiled into a single shell script which is injected as the .Path variable. To quote the Packer <a href="https://www.packer.io/docs/provisioners/shell.html">documentation</a>:</p>
<blockquote>
<p>The -S flag tells sudo to read the password from stdin, which in this case is being piped in with the value of [the user variable ssh_pass.] The -E flag tells sudo to preserve the environment, allowing our environmental variables to work within the script.</p>
<p>By setting the execute_command to this, your script(s) can run with root privileges without worrying about password prompts.</p>
</blockquote>
<p>Taking advantage of this trick, a command can now be placed in the inline section that will add all members of the sudo group to the sudoers file granting them permission to use sudo without a password. Now I know this isn’t secure but for my purpose, which is to create a custom development enviornment on a Virtual Machine running only on my machine, this will be just fine. I also use this example to illustrate how to run commands as sudo.</p>
<h3 id="the-ansible-provisioner">The Ansible Provisioner</h3>
<p>Once the shell provisioner is working, Ansible can be installed on the new machine and then executed using Packer’s Ansible provisioner. The easiest way to do this that I found was to have the shell provisioner install Ansible on the virtual machine as well as upload an ssh public key so that the Ansible user could log in. My “provisioners” section looks like this:</p>
<pre tabindex="0"><code>"provisioners": [
{
"type": "shell",
"inline": [
"mkdir .ssh",
"echo '{{user `public_key`}}' >> .ssh/authorized_keys"
]
},
{
"type": "shell",
"execute_command": "echo '{{user `ssh_pass`}}' | {{ .Vars }} sudo -E -S sh '{{ .Path }}'",
"inline": [
"add-apt-repository ppa:rquillo/ansible",
"apt-get update",
"apt-get install -y ansible",
"echo '%sudo ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers"
]
},
{
"type": "ansible-local",
"playbook_file": "site.yml"
}
]
</code></pre><p>This configuration uses a variable in which I placed an ssh public key (I could also have used the File Uploads provisioner for this), installs Ansible from an updated PPA, and grants the user sudo priveliges via the sudo group as explained above. This also shows how you can execute more than one statement at a time using the “shell” provisioner.</p>
<h3 id="ansible-provisioning-tip">Ansible Provisioning Tip</h3>
<p>This tip could probably apply to any of the devops tools you’d like to use. If you are creating your Ansible yml files for the first time, you will likely run into the issue where you spend a lot of time waiting for the machine to build and provision only to discover your Ansible script is wrong. Troubleshooting becomes a problem because if anything fails during the provision, Packer will stop running and delete the Virtual Machine leaving you with no option other than fixing your mistake and then waiting for the entire process to run again.</p>
<p>One way I found around this is to take the last section of the provisioners array out, build your machine, and then move the base machine into a new directory once it’s been successfully built. From there you can start the machine manually and then run the ansible-playbook command from your local machine while you develop your playbook. Once you have a working playbook, add the ansible-local section back to the provisioners array and rebuild your machine with Packer. That should speed up your development and troubleshooting cycles.</p>
<h3 id="a-hiccup-with-ansible-and-packer">A Hiccup with Ansible and Packer</h3>
<p>Ansible allows you to create template files that can be used for configuration files and the like. According to the documentation, you can specify the location of the templates and other files using the playbook_paths option in the provisioner. I could not get this to work and after a lot of troubleshooting and looking at the code for the provisioner I am convinced there is a bug copying the playbook_paths directories to the remote machine. I’ve <a href="https://groups.google.com/forum/#!topic/packer-tool/RrIGFH3K1bE">posted on the Packer discussion group</a> about this but haven’t had any response on it yet. Once I get to the bottom the issue I’ll post an update here.</p>
<h3 id="conclusion">Conclusion</h3>
<p>Packer has turned out to be a fabulous resource for me in quickly ramping up development environments. Ansible has also been a breath for fresh air for provisioning those machines. I’ve previously used <a href="https://www.getchef.com/chef/">chef</a> and other devops tools which only led to a great deal of frustration. I’m happy to have some new tools in my belt that took very little time to learn and to get working.</p>
Provisioning a Development Environment with Packer, Part 1https://www.endpointdev.com/blog/2014/03/provisioning-development-environment/2014-03-12T00:00:00+00:00Mike Farmer
<p>I recently needed to reconstruct an old development environment for a project I worked on over a year ago. The codebase had aged a little and I needed old versions of just about everything from the OS and database to Ruby and Rails. My preferred method for creating a development environment is to setup a small virtual machine (VM) that mimics the production environment as closely as possible.</p>
<h3 id="introducing-packer">Introducing Packer</h3>
<p>I have been hearing a lot of buzz lately about <a href="https://www.packer.io/">Packer</a> and wanted to give it a shot for setting up my environment. Packer is a small command line tool written in the increasingly popular <a href="https://golang.org/">Go</a> programming language. It serves three primary purposes:</p>
<ol>
<li>Building a machine based on a set of configuration parameters</li>
<li>Running a provisioner to setup the machine with a desired set of software and settings</li>
<li>Performing any post processing instructions on that machine</li>
</ol>
<p>Packer is really simple to install and I would refer you to their <a href="https://www.packer.io/docs/installation.html">great documentation</a> to get it set up. Once set up, you will have the <code>packer</code> command at your disposal. To build a new machine, all you need to is call:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-sh" data-lang="sh">packer build my_machine.json
</code></pre></div><p>The file my_machine.json can be the name of any json file and contains all the information Packer needs to setup your machine. The configuration json has three major sections: variables, builders, and provisioners. Variables are simply key value pairs that you can reference later in the builders and provisioners sections.</p>
<h3 id="the-builder-configuration">The Builder Configuration</h3>
<p><a href="https://web.archive.org/web/20160318014945/https://www.packer.io/docs/templates/builders.html">Builders</a> takes an array of JSON objects that specify different ways to build your machines. You can think of them as instructions on how to get your machine setup and running. For example, to get a machine up and running you need to create a machine, install an operating system (OS) and create a user so that you can login to the machine. There are many different types of builders, but for the example here, I’ll just use the <code>vmware-iso</code> machine type. Here’s a working JSON configuration file:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-json" data-lang="json">{
<span style="color:#b06;font-weight:bold">"variables"</span>: {
<span style="color:#b06;font-weight:bold">"ssh_name"</span>: <span style="color:#d20;background-color:#fff0f0">"mikefarmer"</span>,
<span style="color:#b06;font-weight:bold">"ssh_pass"</span>: <span style="color:#d20;background-color:#fff0f0">"mikefarmer"</span>,
<span style="color:#b06;font-weight:bold">"hostname"</span>: <span style="color:#d20;background-color:#fff0f0">"packer-test"</span>
},
<span style="color:#b06;font-weight:bold">"builders"</span>: [
{
<span style="color:#b06;font-weight:bold">"type"</span>: <span style="color:#d20;background-color:#fff0f0">"vmware-iso"</span>,
<span style="color:#b06;font-weight:bold">"iso_url"</span>: <span style="color:#d20;background-color:#fff0f0">"os/ubuntu-12.04.4-server-amd64.iso"</span>,
<span style="color:#b06;font-weight:bold">"iso_checksum"</span>: <span style="color:#d20;background-color:#fff0f0">"e83adb9af4ec0a039e6a5c6e145a34de"</span>,
<span style="color:#b06;font-weight:bold">"iso_checksum_type"</span>: <span style="color:#d20;background-color:#fff0f0">"md5"</span>,
<span style="color:#b06;font-weight:bold">"ssh_username"</span>: <span style="color:#d20;background-color:#fff0f0">"{{user `ssh_name`}}"</span>,
<span style="color:#b06;font-weight:bold">"ssh_password"</span>: <span style="color:#d20;background-color:#fff0f0">"{{user `ssh_pass`}}"</span>,
<span style="color:#b06;font-weight:bold">"ssh_wait_timeout"</span>: <span style="color:#d20;background-color:#fff0f0">"20m"</span>,
<span style="color:#b06;font-weight:bold">"http_directory"</span> : <span style="color:#d20;background-color:#fff0f0">"preseeds"</span>,
<span style="color:#b06;font-weight:bold">"http_port_min"</span> : <span style="color:#00d;font-weight:bold">9001</span>,
<span style="color:#b06;font-weight:bold">"http_port_max"</span> : <span style="color:#00d;font-weight:bold">9001</span>,
<span style="color:#b06;font-weight:bold">"shutdown_command"</span>: <span style="color:#d20;background-color:#fff0f0">"echo {{user `ssh_pass`}} | sudo -S shutdown -P now"</span>,
<span style="color:#b06;font-weight:bold">"boot_command"</span>: [
<span style="color:#d20;background-color:#fff0f0">"<esc><esc><enter><wait>"</span>,
<span style="color:#d20;background-color:#fff0f0">"/install/vmlinuz noapic "</span>,
<span style="color:#d20;background-color:#fff0f0">"preseed/url=http://{{ .HTTPIP }}:{{ .HTTPPort }}/precise_preseed.cfg "</span>,
<span style="color:#d20;background-color:#fff0f0">"debian-installer=en_US auto locale=en_US kbd-chooser/method=us "</span>,
<span style="color:#d20;background-color:#fff0f0">"hostname={{user `hostname`}} "</span>,
<span style="color:#d20;background-color:#fff0f0">"fb=false debconf/frontend=noninteractive "</span>,
<span style="color:#d20;background-color:#fff0f0">"keyboard-configuration/modelcode=SKIP keyboard-configuration/layout=USA "</span>,
<span style="color:#d20;background-color:#fff0f0">"keyboard-configuration/variant=USA console-setup/ask_detect=false "</span>,
<span style="color:#d20;background-color:#fff0f0">"initrd=/install/initrd.gz -- <enter>"</span>
]
}
]
}
</code></pre></div><p>The <a href="https://web.archive.org/web/20160318014945/https://www.packer.io/docs/templates/builders.html">documentation</a> for these settings is really good but I want to point out a few things that weren’t immediately clear. Some of these pertain mostly to the vmware-iso builder type, but I believe they are worth pointing out because some of them apply to other builder types as well.</p>
<p>First, the <code>iso_url</code> setting can be either an absolute path, a relative path, or a fully qualified url. The relative path is relative to the directory where you run the <code>packer</code> command. So here, when I run <code>packer</code>, I need to make sure that I do so from a directory that has an os subdirectory with the ubuntu iso located therein.</p>
<p>Next, once the ISO is downloaded, Packer will automatically start up your VMWare client and boot the virtual machine. Immediately after that, Packer will start up a <a href="https://en.wikipedia.org/wiki/Virtual_Network_Computing">VNC client and server</a> along with a mini web server to provide information for your machine. The <code>http_port_min</code> and <code>http_port_max</code> specify which ports to use for the VNC clients. Setting them to the same will allocate just that port for it to use. The <code>http_directory</code> setting provides the name of a local directory to use for the mini web server as the document root. This is important for providing your VM with a preseed file, more about the preseed file will be discussed below.</p>
<p>Since we are using Ubuntu as our main machine, we will need to use sudo to send the shutdown command. The <code>shutdown_command</code> setting is used to gracefully shut down the machine at the conclusion of the run and provisioning of the machine.</p>
<h3 id="installing-your-os">Installing your OS</h3>
<p>The <code>boot_command</code> is a series of keystrokes that you can send to the machine via VNC. If you have set up a Linux machine from scratch you know that you have to enter in a bunch of information to the machine about how to set it up for the first time such as time zone, keyboard layout, how to partition the hard drive, host name, etc. All these keystrokes needed to set up your machine can be used here.</p>
<p>But if you think about it, that’s a ton of keystrokes and this command could get quite long. A better way to approach this is to use a preseed file. A <a href="https://web.archive.org/web/20210518144703/https://help.ubuntu.com/lts/installation-guide/s390x/apb.html">preseed.cfg</a> file contains the same information you enter when you setup a machine for the first time. This isn’t something provided by Packer, but it is provided by the operating system to automatically provision machines. For Ubuntu, a preseed file is used like so:</p>
<ul>
<li>When you boot from the startup media (in this case an ISO image), you can choose the location of the preseed file via a URL.</li>
<li>The preseed file is uploaded into memory and the configuration is read.</li>
<li>The installation process begins using information from the preseed file to enter the values where the user would normally enter them.</li>
</ul>
<p>So how do we get the preseed file up to the machine? Remember that little web server that Packer sets up? Well, the IP address and port is made available to the virtual machine when it boots from the ISO. The following line tells the OS where to find the web server and the configuration file:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-json" data-lang="json"><span style="color:#d20;background-color:#fff0f0">"preseed/url=http://{{ .HTTPIP }}:{{ .HTTPPort }}/precise_preseed.cfg"</span>
</code></pre></div><p>Strings in Packer can be interpolated using a simple template format similar to <a href="https://mustache.github.io/">mustache</a>. The double curly braces tell Packer to insert a variable here instead of text. The HTTPIP and HTTPPort variables are made available by Packer to the template.</p>
<p>One more important note about the preseed file: You need to make sure that the settings for the username and password are the same as listed in your variables section so that you can login to the machine once it is built. Where do you get a preseed file? I found <a href="https://kappataumu.com/articles/creating-an-Ubuntu-VM-with-packer.html">one on a blog titled Packer in 10 Minutes</a> by <a href="https://twitter.com/kappataumu">@kappataumu</a>. I only had to modify a few settings that were specific to my setup.</p>
<p>Remember that <code>http_directory</code> mentioned above? Well, that directory needs to include your preseed file. I’ve named mine <code>precise_preseed.cfg</code> for Ubuntu 12.04 Precise Pangolin.</p>
<p>Next up is provisioning but that is such a big topic by itself that I’ll move that into a separate blog post. The config file above will work as-is and once run it should setup a basic Ubuntu server for you. Go ahead and give it a try and let me know in the comments how it worked out for you.</p>
<h3 id="super-powers">Super Powers</h3>
<p>I said that Packer has 3 primary purposes earlier. Well, I lied.</p>
<p>Packer’s superpower is that it can perform those 3 purposes over any number of machines, whether virtual, hosted, or otherwise in parallel. Supported machines are currently:</p>
<ul>
<li>Amazon EC2 (AMI)</li>
<li>DigitalOcean</li>
<li>Docker</li>
<li>Google Compute Engine</li>
<li>OpenStack</li>
<li>QEMU</li>
<li>VirtualBox</li>
<li>VMware</li>
</ul>
<p>Consider for a moment that you can now automatically setup and provision multiple machines with the same environment using a single command. Now you are seeing the power of Packer.</p>
Monitoring Productivity with RescueTimehttps://www.endpointdev.com/blog/2014/02/monitoring-productivity-with-rescuetime/2014-02-26T00:00:00+00:00Mike Farmer
<div class="separator" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em; text-align: center;">
<a href="http://rescuetime.com/"><img alt="RescueTime.com" border="0" src="/blog/2014/02/monitoring-productivity-with-rescuetime/image-0.png" title="RescueTime"/></a></div>
<p>I’m always on the lookout for tools that help me be more productive in my daily work. Time tracking is always one of those difficult tasks for those who bill time by the hour. I’ll admit that there are days here and there where I know I’ve worked all day but at the end of the day I can’t remember where I spent my time and how long I’ve spent on different tasks. Last week I discovered a new tool that I have loved! It’s called <a href="http://www.rescuetime.com/">RescueTime</a> and it has changed how I approach my days.</p>
<p>RescueTime is a web service that you can sign up for free. You download a little app that runs in the background and monitors your productivity throughout the day. You can then log into the RescueTime website and see many different breakdowns of how you spent your day.</p>
<p>So how does it track productivity? It monitors how much time you spend in different applications on your computer. Then it tries to tag that time on a scale from distracting to very productive. You might be thinking that if you spend your whole day in a browser that wouldn’t be very helpful. But RescueTime is pretty smart and it also monitors which web sites you visit and lets you categorize that time as well.</p>
<div class="separator" style="clear: both; text-align: center;">
<a href="/blog/2014/02/monitoring-productivity-with-rescuetime/image-0.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="141" src="/blog/2014/02/monitoring-productivity-with-rescuetime/image-0.png" width="320"/></a></div>
<p>The nicest thing about RescueTime is how little time you spend actually tending the app. It’s always there and their automatic categorization gets it right almost all the time. I simply love logging into the dashboard on the site at the end of the day and seeing how my time has been spent that day. So far it has helped motivate me to spend as little time on the distracting items of my day and focus on what is truly productive and important.</p>
<div class="separator" style="clear: both; text-align: center;">
<a href="/blog/2014/02/monitoring-productivity-with-rescuetime/image-1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="256" src="/blog/2014/02/monitoring-productivity-with-rescuetime/image-1.png" width="640"/></a></div>
<p>RescueTime also gives you the opportunity to set goals for yourself and tracks your progress over time. I haven’t used these features yet, but I plan to once I’ve used the app long enough to collect enough information about my days to better inform my goal setting. Lifehacker.com has a great article no how to optimize your time using RescueTime called “<a href="http://lifehacker.com/5839691/how-to-reclaim-your-time-in-seven-days-with-rescuetime">How to Reclaim Your Time in Seven Days for a More Productive and Stress-Free Workday</a>” and I’d recommend giving it a quick read.</p>
<p>RescueTime also provides a premium service that not just monitors but actually interacts with your computer as well. For example, you can tell it to block certain web sites during certain times of the day. For what I’m doing that’s a bit overkill but I can see how the value could play out if you’re really having a hard time kicking some distracting online habit.</p>
Mobile Emulation in Chrome DevToolshttps://www.endpointdev.com/blog/2014/01/mobile-emulation-in-chrome-devtools/2014-01-24T00:00:00+00:00Greg Davidson
<p>I have been doing some mobile development lately and wanted to share the new Mobile Emulation feature in Chrome Canary with y’all. <a href="https://www.google.ca/intl/en/chrome/browser/canary.html">Chrome Canary</a> is a development build of Chrome which gets updated daily and gives you the chance to use the latest and greatest features in Chrome. I’ve been using it as my primary browser for the past year or so and it’s been fairly stable. What’s great is that you can run Chrome Canary side-by-side with the stable release version of Chrome. For the odd time I do have issues with stability etc., I can just use the latest stable Chrome and be on my way. If you need more convincing, Paul Irish’s <a href="http://www.paulirish.com/2012/chrome-canary-for-developers/">Chrome Canary for Developers</a> post might be helpful.</p>
<p>I should mention that Chrome Canary is only available for OS X and Windows at this point. I tested <a href="http://www.chromium.org/getting-involved/dev-channel#TOC-Linux">Dev channel Chromium</a> on Ubuntu 13.10 this afternoon and the new mobile emulation stuff is not ready there yet. It should not be long though.</p>
<h3 id="mobile-emulation-in-chrome-dev-tools">Mobile Emulation in Chrome Dev Tools</h3>
<img alt="Mobile emulation chrome canary" border="0" height="334" src="/blog/2014/01/mobile-emulation-in-chrome-devtools/image-0.png" title="mobile-emulation-chrome-canary.png" width="559"/>
<p>Once enabled, the Emulation panel shows up in the Dev Tools console drawer. It gives you the option of emulating a variety devices (many are listed in the drop-down) and you also have the ability to fine tuning the settings à la carte. If you choose to emulate the touchscreen interface the mouse cursor will change and operate like a touch interface. Shift+drag allows you to pinch and zoom. There are some cool features for debugging and inspecting touch events as well.</p>
<h3 id="learning-more">Learning More</h3>
<p>If you would like to learn more, be sure to check out the <a href="https://developers.google.com/chrome-developer-tools/docs/mobile-emulation">Mobile emulation documentation</a> at the Chrome DevTools docs site.</p>
Copy Data Between MySQL Databases with Sequel Prohttps://www.endpointdev.com/blog/2014/01/copy-data-between-mysql-databases-with/2014-01-10T00:00:00+00:00Greg Davidson
<h3 id="sequel-pro">Sequel Pro</h3>
<img alt="Sequel pro" border="0" height="350" src="/blog/2014/01/copy-data-between-mysql-databases-with/image-0.png" style="display:block" title="sequel-pro.png" width="450"/>
<p>I often use <a href="https://www.sequelpro.com/">Sequel Pro</a> when I’m getting up to speed on the data model for a project or when I just want to debug in a more visual way than with the mysql command-line client. It’s a free OS X application that lets you inspect and manage MySQL databases. I also find it very useful for making small changes to the data while I develop and test web apps.</p>
<h3 id="quickly-copy-data-between-databases">Quickly Copy Data Between Databases</h3>
<p>I recently needed a way to copy a few dozen records from one <a href="http://www.devcamps.org/">camp</a> to another. I tried using the <a href="https://dev.mysql.com/doc/refman/5.7/en/select-into.html">“SELECT…INTO OUTFILE”</a> method but ran into a permissions issue with that approach. Using <a href="https://dev.mysql.com/doc/refman/5.7/en/mysqldump.html">mysqldump</a> was another option but that seemed like overkill in this case—I only needed to copy a few records from a single table. At this point I found a really neat and helpful feature in Sequel Pro: Copy as SQL INSERT</p>
<img alt="Copy as sql insert" border="0" height="407" src="/blog/2014/01/copy-data-between-mysql-databases-with/image-1.png" title="copy-as-sql-insert.png" width="562"/>
<p>I simply selected the records I wanted to copy and used the “Copy as SQL INSERT” feature. The SQL insert statement I needed was now copied to the system clipboard and easily copied over to the other camp and imported via the mysql command-line client.</p>
<h3 id="bundles">Bundles</h3>
<p>The Sequel Pro website describes <a href="https://www.sequelpro.com/bundles">Bundles</a> which extend the functionality in various ways—including copying data as JSON. Very handy stuff. Many thanks to the <a href="https://web.archive.org/web/20140208071143/http://northofthree.com/">developers</a> of this fine software. If you’re on OS X, be sure to give it a try.</p>