Data Center Relocation Initiatives: Daunting But Achievable

2018-12-20T00:00:00+00:00

Introduction

A customer asked for our help dealing with logistical nightmares they encountered during a hardware update and data center relocation project. The customer had two active data centers, and wanted to relocate one of them to a modern Tier 4 facility to improve its performance and provide redundancy for critical systems. They also wanted to consolidate or decommission equipment to reduce recurring expenses and reduce their carbon footprint. The client decided to move to a Tier 4 data center because they provide across-the-board redundancy within the data center.

Moving forward with Tier IV

Enterprises typically transition to Tier 4 data centers because they offer the highest uptime guarantees, and have no single points of failure. These facilities are fully redundant in terms of electrical circuits, cooling, and networking. This architecture is best able to withstand even the most serious technical incidents without server availability being affected. Tier IV facilities have contracts with disaster management companies who will provide them, for example, with fuel in the event that a natural disaster damages the power grid.

(More information about the four data center levels is at the end of this post.)

Key Issues

Some of the customer’s key concerns were:

1. To safely move their virtual environment to the new data center.

2. To protect assets during the relocation.

3. To completely shut down the current data center and migrate seamlessly to the new data center.

4. To update external DNS records without causing any downtime for the web applications used by their customers.

Project Planning

To manage data center relocations is challenging because there are many moving parts and variables. Typically, planning starts 6 months ahead of the scheduled relocation date. We seek to understand the role of each system by consulting in-depth with all the teams involved. We analyze each part of the client’s tech infrastructure, break it down, and strategize about the best approach for handling the migration of each component. Additional time to plan can help especially when a large group of stakeholders is involved. Successful planning requires hard and sometimes extensive work, but makes an otherwise overwhelming data center migration smooth and worry free.

Here are some of our procedures for pre-migration preparation:

1. Process the paperwork for new hardware. Identify, review, and approve all applications. Hire movers. Schedule and coordinate with internal management and key stakeholders like IT staff, system admins, contractors, circuit providers, etc.

2. Collect and analyze resource usage associated with the critical and non-critical systems and compare resource needs to resource availability at the new facility.

3. Design and schedule the migration of critical systems to the new data center. Discuss the plan with key stakeholders.

4. Make sure replicas of critical systems are up to date.

5. Set up a new environment with new hardware in the new data center and establish network connections.

6. Prepare hardware and virtualization technology in the new data center.

7. Test, retest, and burn in the new hardware (2–4 weeks of testing).

8. Create a non-essential server within the virtualization environment and test it.

A Challenge: Eliminating Downtime

To minimize or eliminate downtime requires very careful preparation. With this particular client, we had the luxury of their already having a secondary data center, which could accommodate a majority of the critical systems during the migration.

We originally planned to leverage this by simply changing the external DNS records to point to this data center during the migration, but during planning determined that the external DNS record would not replicate worldwide quickly enough to meet the uptime needs of their global operation.

Solutions

After discussing various approaches, we ended up using a load balancer. A load balancer is a network device that routes incoming traffic destined for a single destination (web site, application, or service) and ‘shares’ the incoming connections across multiple destination devices or services. In this case, the data center we were to migrate to housed the client’s critical systems like web servers, database, etc. The secondary data center housed a replica of the critical systems using a replication technology, e.g. Actifio, Rubrik, SAN mirroring, VMWare SRM, or Zerto — we used Actifio in this scenario.

The customer had multiple systems with external DNS records and public IP addresses. We did not change the public DNS names for the critical systems. We did change the public IP addresses associated with the public DNS names, to point to the load balancer, which then pointed to the appropriate data center throughout the process. This setup enabled a gradual, flexible, and fully controlled transition to the new data center.

Summary

Our client wanted to reduce their servers’ power usage 40% by consolidating into a hyper-converged system. Other goals were to reduce floor space by 50%, and monthly recurring data center charges by 50%. We transitioned them to a new hardware stack (compute, storage, network, backup environment) in a Tier 4 data center, and saved them 50% percent on data center operation costs. Additionally, during the process we corrected issues associated with disaster recovery and business continuity, leaving the client’s operations more resilient, affordable, and more secure.

Appendix: Data Center Classifications

Tier IV (Best)

Tier IV data centers have redundancies for every process and data protection stream. No single outage or error can shut down the system. To keep Tier IV ranking, data centers must have 99.995% minimum uptime per year. They come equipped with 2N+1 infrastructure (two times the amount required for operation plus a backup), which is considered “fully redundant.”

Tier III (Average)

N+1 (the amount required for operation plus a backup) fault tolerance, which means a Tier III facility can undergo routine maintenance without a hiccup in operations. 99.982% minimum uptime per annum, used for maintenance and overwhelming emergencies.

Tier II (Minimal)

99.749% minimum uptime. Tier II facilities have considerably more downtime than Tier III facilities, primarily because Tier II facilities have less redundancy. They do however have partial cooling redundancy and multiple power redundancies.

Tier I (Basic, Cheapest)

99.671% minimum uptime, with 28.8 hours of downtime each year.

Shell Command Outputs Truncated in Python

2018-04-05T00:00:00+00:00

Photo by Sarah Pflug of Burst

Recently I was working on a Python script to do some parsing and processing on the output of shell commands in Ubuntu. The output that showed up was truncated.

The below sections will walk through the debugging process to identify the root cause and implement a solution with detailed explanation, using Python 2.

Problem

The following code block shows the output of a shell command which lists the installed packages, name and version, in Ubuntu.

# dpkg -l | grep ^ii | awk '{print $2 "    " $3}'
accountsservice    0.6.35-0ubuntu7.3
acl    2.2.52-1
adduser    3.113+nmu3ubuntu3
ant    1.9.3-2build1
ant-optional    1.9.3-2build1
apache2    2.4.7-1ubuntu4.18
apache2-bin    2.4.7-1ubuntu4.18
apache2-data    2.4.7-1ubuntu4.18
apache2-utils    2.4.7-1ubuntu4.18
apparmor    2.10.95-0ubuntu2.6~14.04.1

The same shell command executes in the Python console but the output shows truncated values for a few packages’ versions, for example, accountsservice, adduser, apache2, etc.

>>> import subprocess
>>> installed_packages = subprocess.check_output(['dpkg -l | grep ^ii | awk \'{print $2 "    " $3}\''], shell=True)
>>> print installed_packages
accountsservice    0.6.35-0ubuntu7.
acl    2.2.52-1
adduser    3.113+nmu3ubuntu
ant    1.9.3-2build1
ant-optional    1.9.3-2build1
apache2    2.4.7-1ubuntu4.1
apache2-bin    2.4.7-1ubuntu4.1
apache2-data    2.4.7-1ubuntu4.1
apache2-utils    2.4.7-1ubuntu4.1
apparmor    2.10.95-0ubuntu2

Root Cause

To identify the root cause of the problem, I started with source command dpkg -l command without any filters and processing. I have noticed two different results for this command, with and without less command. The less command showed the complete result with scrolling as below.

# dpkg -l | less
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                                  Version                                    Architecture Description
+++-=====================================-==========================================-============-===============================================================================
rc  aacraid                               1.2.1-52011                                amd64        This driver supports Adaptec by PMC aacraid family of cards.
ii  accountsservice                       0.6.35-0ubuntu7.3                          amd64        query and manipulate user account information
ii  acl                                   2.2.52-1                                   amd64        Access control list utilities
ii  adduser                               3.113+nmu3ubuntu3                          all          add and remove users and groups
ii  ant                                   1.9.3-2build1                              all          Java based build tool like make
ii  ant-optional                          1.9.3-2build1                              all          Java based build tool like make - optional libraries
ii  apache2                               2.4.7-1ubuntu4.18                          amd64        Apache HTTP Server
ii  apache2-bin                           2.4.7-1ubuntu4.18                          amd64        Apache HTTP Server (binary files and modules)
ii  apache2-data                          2.4.7-1ubuntu4.18                          all          Apache HTTP Server (common files)
ii  apache2-utils                         2.4.7-1ubuntu4.18                          amd64        Apache HTTP Server (utility programs for web servers)
rc  apache2.2-common                      2.2.22-1ubuntu1.11                         amd64        Apache HTTP Server common files
ii  apparmor                              2.10.95-0ubuntu2.6~14.04.1                 amd64        user-space parser utility for AppArmor

But dpkg -l prints on the screen with truncated data due to the columns width constraint. The truncated values exactly match the Python console output. The output column width is decided by environment variable COLUMNS and commands restrict the column width in output based on COLUMNS value.

# echo $COLUMNS
127

# dpkg -l
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                   Version          Architecture     Description
+++-======================-================-================-==================================================
rc  aacraid                1.2.1-52011      amd64            This driver supports Adaptec by PMC aacraid family
ii  accountsservice        0.6.35-0ubuntu7. amd64            query and manipulate user account information
ii  acl                    2.2.52-1         amd64            Access control list utilities
ii  adduser                3.113+nmu3ubuntu all              add and remove users and groups
ii  ant                    1.9.3-2build1    all              Java based build tool like make
ii  ant-optional           1.9.3-2build1    all              Java based build tool like make - optional librari
ii  apache2                2.4.7-1ubuntu4.1 amd64            Apache HTTP Server
ii  apache2-bin            2.4.7-1ubuntu4.1 amd64            Apache HTTP Server (binary files and modules)
ii  apache2-data           2.4.7-1ubuntu4.1 all              Apache HTTP Server (common files)
ii  apache2-utils          2.4.7-1ubuntu4.1 amd64            Apache HTTP Server (utility programs for web serve
rc  apache2.2-common       2.2.22-1ubuntu1. amd64            Apache HTTP Server common files
ii  apparmor               2.10.95-0ubuntu2 amd64            user-space parser utility for AppArmor

Solution

The subprocess module of Python provides complete untruncated output of the shell command when the argument env={} is passed to check_output function:

>>> installed_packages = subprocess.check_output(['dpkg -l | grep ^ii | awk \'{print $2 "    " $3}\''], shell=True, env={})
>>> print installed_packages
accountsservice    0.6.35-0ubuntu7.3
acl    2.2.52-1
adduser    3.113+nmu3ubuntu3
ant    1.9.3-2build1
ant-optional    1.9.3-2build1
apache2    2.4.7-1ubuntu4.18
apache2-bin    2.4.7-1ubuntu4.18
apache2-data    2.4.7-1ubuntu4.18
apache2-utils    2.4.7-1ubuntu4.18
apparmor    2.10.95-0ubuntu2.6~14.04.1

Explanation

Curious to know what is happening behind the scenes? The check_output function uses C library functions execv or execve for processing. It chooses the function based on the env argument.

Reference:

When no env argument is passed to subprocess.check_output, the os.execv function is called.

When an env argument is passed to subprocess.check_output, the os.execve function is called.

for (i = 0; exec_array[i] != NULL; ++i) {
    const char *executable = exec_array[i];
    if (envp) {
        execve(executable, argv, envp);
    } else {
        execv(executable, argv);
    }

What makes the execv and execve functions produce different output?

The execv function passes through the shell COLUMNS variable which leads to truncating output columns to 127 width, like our reference system.

# echo $COLUMNS
127

>>> print subprocess.check_output(['dpkg -l | grep libqtcore4'], shell=True)
ii  libqtcore4:amd64          4:4.8.5+git192-g0 amd64             Qt 4 core module

>>> print subprocess.check_output(['dpkg -l | grep libqtcore4'], shell=True, env={'COLUMNS':'127'})
ii  libqtcore4:amd64          4:4.8.5+git192-g0 amd64             Qt 4 core module

The execve function uses additional argument environment variables and it is based on the environ function. It uses environment variables available in env command which doesn’t have COLUMNS initialised. So output values returned without any column width restriction.

>>> print subprocess.check_output(['dpkg -l | grep libqtcore4'], shell=True, env={})
ii  libqtcore4:amd64                      4:4.8.5+git192-g085f851+dfsg-2ubuntu4.1    amd64        Qt 4 core module

>>> print subprocess.check_output(['dpkg -l | grep libqtcore4'], shell=True, env={'COLUMNS':''})
ii  libqtcore4:amd64                      4:4.8.5+git192-g085f851+dfsg-2ubuntu4.1    amd64        Qt 4 core module

For more details refer to the man pages of execv, execve, environ.

Conclusion

It is always good to pass env={} argument to subprocess.check_output function whenever processing shell command output in Python. It helps avoid unstable results down the line due to truncated values.

Separate and Not Equal: Development Environments to Support People, Process, and Automation

2017-01-18T00:00:00+00:00

I believe in the separation of software environments for the long term peace and prosperity of developers, product owners, and users. Ideally, there’s a dedicated environment purpose-built to support each community: the developers, the testers, the acceptors, and the users.

I learned this process from taking part in software development projects over the course of many years and working with a variety of different teams, including my own dev team, our clients’ in-house IT teams, and often other consulting firms. While each group had its own approach, one consistent factor for success was the separation of environments to support key roles.

Dev

Development environments will vary by project and technology stack. A common decision is developing locally or creating a remote VM, and both approaches have trade-offs. Developing remotely allows you to setup an environment that mirrors QA and Production, and it also allows you to use remote desktop to connect to your dev environment from any machine: PC, Mac, or Linux. However, developing remotely comes with latency, and often it doesn’t feel as smooth as working locally. More importantly, if I find myself traveling or otherwise without a stable internet connection, developing remotely can be challenging if not impossible.

The setup of the dev environment should be quick and painless, and the project should have clear documentation on how to get started developing. We often keep this in a Wiki in the project’s Git repository. Pairing this documentation with a pre-built dev environment is the ultimate way to get new developers up-and-running. This can be done by packaging up a VM, which can then be run locally or remotely using virtualization software like VirtualBox or a cloud platform like AWS. Vagrant is another option worth checking out, as it touts a one-command setup of a complete dev environment.

QA

The QA environment serves two purposes: as a place where various team members’ code can be built and run together, and a place where the specific task of QA can be performed. When you move to QA testing, it’s a good idea to create a separate branch in source control.This allows developers to push to a common QA branch, which can then be deployed to the QA environment, ideally using a CI tool like TeamCity.

Once the QA testers have signed off, you can move into the UAT/Staging environment. It’s crucial that this environment match production as closely as possible, to rule out any “environmental bugs” that may creep in after deployment to Production. This means ensuring that all the framework versions match (.NET, Node, etc.), system resources are similar, and that the same security is in place (firewalls, VPNs). The data should also replicate what will be found in production, if not outright synced from production using a tool like SQL Delta.

Stage

In staging, the software is put before a specially designated group of users for a dress rehearsal.

If I’m developing a desktop or mobile application (not a web app), I ensure that testing on a variety of different OS versions is incorporated into the UAT process. For example, I use Apple’s TestFlight to test across a variety of devices using different iOS versions.

Prod

Production is the main stage, and code should obviously only be pushed there after proper testing. Even if I have customers, clients, and/or bosses yelling at me, I try and resist the urge to push a fix directly to Production! Although the fix may have worked perfectly in my dev environment, it might not work in production, and then I can easily find myself in even deeper trouble than before. If there is not time to coordinate UAT, then at the very least I push to QA, where the build can be verified in a separate environment, and another set of eyes can perform some testing. It goes without saying that all of this can be made less frightful with unit testing, a topic that we will explore later.

It’s also important to ensure that production has adequate resources before going live. If I’m building a new version or replacement of existing software, I review the existing reporting/analytics to know things like concurrent user count and storage requirements per user. This may seem obvious, but in the rush to get software out to market, basic things can be overlooked. Also, I ensure that I have some baseline security in place on production. This means encrypting sensitive data in config files, not storing passwords (just salted hashes!), and if I’m using a cloud provider like AWS or Azure, I add two-factor authentication to the admin account.

8 Simple Steps to Saner Software Development

2016-11-10T00:00:00+00:00

While these might seem obvious for seasoned developers, many projects, especially legacy software, still fail to follow most of the steps below.

In future posts, I’ll expand on these steps and give more details. For now, here’s an overview of some basic guidelines for making the process of software development smoother (& saner) for both the client and the dev team.

Separate environments:

Ideally there should be Dev, QA, UAT/Staging, and Production environments. And, UAT/Staging should be as close to Production as possible. It’s amazing how many software projects are still done on someone’s local machine and pushed directly to Production.

Use a bug tracking tool:

This is pretty obvious. Bugs need to be logged and labeled so that they can be tracked through stages of development (i.e., To-Do, In Progress, Completed). Using a bug tracking tool like Jira or GitHub also helps the non-technical stakeholders comment on and clarify requirements/issues.

Have a source control strategy:

It doesn’t have to be fancy. For example, if you’re using Git, it should be more than having multiple developers working out of the master branch. Ideally link branches to features or bugs defined in a bug tracking tool, and keep in mind the separation of environments, especially if the application is already in Production.

Do pull/merge requests:

Even a short five minute review will improve the overall quality of code, reduce bugs, and communicate changes between developers responsible for different parts of the application. Web-based source control systems like GitLab, GitHub, and Bitbucket make this simple.

Write unit tests:

A lot of developers either feel pressure to move quickly or get lazy and don’t write tests. Regression testing becomes a nightmare without unit tests. Plus it makes you feel better to see all those tests passing.

Use a continuous integration tool:

TeamCity, Jenkins, or even basic GitHooks are some options. Automated deployments save time and, if integrated with unit tests, reduce the bugs that make it to Production.

Setup notifications:

If the CI tool alerts team members of deployment status, hundreds of “is the deployment done yet?” emails and chat messages can be saved. Slack integration is a good option here.

Consolidate project credentials:

Have a secure and centralized location for managing passwords. We like keeping a KeePass checked into the relevant project’s repo.

2015 Perl Dancer Conference videos

2015-12-30T00:00:00+00:00

The 2015 Perl Dancer Conference has recently released the presentation videos. This year the conference was hosted in beautiful Vienna, Austria. Josh Lavin and I were both honored to attend the conference as well as give talks. Earlier, Josh wrote summaries of the conference:

Conference Recap

Conference Presentations

SpaceCamps “The Final Frontier”

I gave a talk exploring new technologies for End Point’s own DevCamps development tool. During the presentation I detailed my research into containers and what a cloud-based development environment might look like.

SpaceCamps Presentation Video

AngularJS & Dancer for Modern Web Development

Josh detailed his experience migrating legacy applications utilizing Dancer, AngularJS, and modern Perl techniques. Josh highlighted the challenges he faced during the process, as well as lessons he learned along the way.

AngularJS & Dancer for Modern Web Development Presentation Video

Lightning Talks

Josh and I both gave short “lightning talks.” Josh’s was on Writing Unit Tests for a Legacy App (Interchange 5), and mine was on Plack & Interchange 5.

To review the rest of the presentations please checkout the Perl Dancer Conference YouTube channel.

Summary

The Perl Dancer community continues to flourish and the conference this year hosted a record 5 core Dancer developers. Dancer is about to release the finalized version of its long awaited plugin infrastructure for Dancer2. A lot of work on this was completed during the conference. Being an organizer of the conference, it brings me great joy to see this success. This news along with the release of Perl 6, I am certain 2016 will be a wonderful year for not only Dancer but the entire Perl community.

Handling databases in dev environments for web development

2015-04-21T00:00:00+00:00

One of the biggest problems for web development environments is copying large amounts of data. Every time a new environment is needed, all that data needs to be copied. Source code should be tracked in version control software, and so copying it should be a simple matter of checking it out from the code repository. So that is usually not the problem. The main problem area is database data. This can be very large, take a long time to copy, and can impact the computers involved in the copy (usually the destination computer gets hammered with IO which makes load go high).

Often databases for development environments are created by copying a database dump from production and then importing that database dump. And since database dumps are text, they can be highly compressed, which can result in a relatively small file to copy over. But the import of the dump can still take lots of time and cause high load on the dev computer as it rebuilds tables and indexes. As long as your data is relatively small, this process may be perfectly acceptable.

Your database WILL get bigger

At some point though your database will get so big that this process will take too long and cause too much load to be acceptable.

To address the problem you can try to reduce the amount of data involved by only dumping a portion of the database data instead of all of it, or possibly using some “dummy sample data” instead. These techniques may work if you don’t care that development environments no longer have the same data as production. However, one serious problem with this is that a bug or behavior found in production can’t be replicated in a development environment because the data involved isn’t the same. For example, say a customer can’t checkout on the live site but you can’t replicate the bug in your development environment to fix the bug. In this example, the root cause of the problem could be a bug in the code handling certain products that are out of stock, and since the dev database didn’t have the same data it could make finding and fixing these types of problems a lot harder.

Snapshots

Another option is to use file system snapshots, like LVM snapshots, to quickly make clones of the database without needing to import the database dump each time. This works great if development environments live on the same server, or at least the development databases live on the same server. You would need to create a volume to hold a single copy of the database; this copy would be the origin for all snapshots. Then for each development environment, you could snapshot the origin volume, mount it read-write in a place accessible by the developer, customize the database configuration (like setting a unique port number to listen on), and then start up the database. This then provides a clone of the entire database in a tiny fraction of the time and uses less disk space and other system resources too.

In using snapshots there are some things you’ll need to be careful about. Snapshots are usually created using copy-on-write tables. The more snapshots mounted read-write, the more IO overhead is involved for the volumes involved. For this reason it is important that writes to the origin volume be avoided as much as possible while the snapshots are open. Also, snapshots that get a lot of writes can fill up their copy-on-write table, and depending on the file system and database that you are using this can be a big problem. So it is important to monitor each open snapshot for how full it is and increase their size if needed so they don’t fill up. Updating the origin database will require shutting down and removing all snapshots first, then update the origin database, then create and mount all the snapshots again. This is because all the copy-on-write tables would get full if you tried to update the origin while the snapshots are open.

Using snapshots like this may sound more complicated, and it is, but the processes involved can be scripted and automated and the benefits can be pretty significant if you have several developers and a lot of data to copy.

On End Point’s Development Environment

2015-03-11T00:00:00+00:00

A few recent conversations have sparked my interest in writing up a blog post that summarizes the familiar elements of our development environments. The majority of End Pointers work remotely, but many of the tools listed below are common to many developers.

ssh/sftp: We do primarily remote development. Many of us are familiar with local development, but remote development with camps (see next point) is typically the most efficient arrangement in working with multiple development instances that are accessible to clients for testing and staging.
camps: DevCamps are a tool specific to and created by End Point, which are development instances with an entire webserver, database, and app server stack, similar to containers like Docker. Check out the DevCamps website for more information.
vim/emacs/nano: While most of our employees use vim or emacs for command-line editors, nano is an inefficient but easy to use editor that we can suggest to new developers. Not many of us use IDEs, if at all.
screen/tmux: screen and tmux are our preferred terminal multitasking and sharing.
command-line database interaction (specifically psql and mysql ad-hoc querying): Working with an SQL database through an ORM like Active Record, DBIC, etc. is not enough for us.
*nix / basic command-line interaction: This topic could make up its own blog post, but some of the tools we use frequently are netstat/ss, ifconfig/ip, lsof, ps/top/htop/atop, free, df, nice/ionice, tail -f, sort, uniq -c, grep.
git & github: Not uncommon to devshops these days, git is the most popular version control system, and github an extremely popular host of both open source and private repositories.
IRC, Skype, Google Hangouts, appear.in, talky.io, glideroom.com, Google Voice, & gasp regular phones: As remote developers, we communicate often and there are a number of tools available in the communication space that we leverage.

It’s interesting to note that if any new developers come in with a preference for a trendy new tool, while we are happy to let them work in an environment that allows them to be efficient, ultimately we can’t provide support for those tools that we are unfamiliar with.

Looking at development environments with DevCamps and Vagrant

2014-08-25T00:00:00+00:00

For most web developers, you have practices and tools that you are used to using to do your work. And for most web developers this means setting up your workstation with all the things you need to do your editing, compiling, testing, and pushing code to some place for sharing or deployment. This is a very common practice even though it is fraught with problems- like getting a database setup properly, configuring a web server, any other services (memcached, redis, mongodb, etc), and many more issues.

Hopefully at some point you realize the pain that is involved in doing everything on your workstation directly and start looking for a better way to do web development. In this post I will be looking at some ways to do this better: using a virtual machine (VM), Vagrant, and DevCamps.

Using a VM for development

One way to improve things is to use a local virtual machine for your development (for example, using VirtualBox, or VMware Fusion). You can edit your code normally on your workstation, but then execute and test it in the VM. This also makes your workstation “clean”, moving all those dependencies (like a database, web server, etc.) off your workstation and into the VM. It also gets your dev environment closer to production, if not identical. Sounds nice, but let’s break down the pros and cons.

Benefits of using a VM

Dev environment closely matches production.
Execute and test code in a dedicated machine (not your workstation directly).
Allows for multiple projects to be worked on concurrently (one VM per project).
Exposes the developer to the Operations (systems administration) side of the web application (always a good thing).
Developer can edit files using their favorite text editor locally on the workstation (but will need to copy files to the VM as needed).

Problems with using a VM

Need to create and configure the VM. This could be very time consuming and error prone.
Still need to install and configure all services and packages. This could also be time consuming and error prone.
Backups of your work/configuration/everything are your own responsibility (extremely unlikely to happen).
Access to your dev environment is extremely limited, thus probably only you can access it and test things on it. No way for a QA engineer or business owner to test/demo your work.
Inexperienced developers can break things, or change them to no longer match production (install arbitrary packages, different versions than what is in production, screw up the db, screw up Apache configuration, etc.).
If working with an established database, then downloading a dump, installing, and getting the database usable is time consuming and error prone. (“I just broke my dev database!” can be a complete blocker for development.)
The developer needs to set up networking for the VM in order to ssh to it, copy files back and forth, and point a web browser to it. This may include manually setting up DNS, or /etc/hosts entries, or port forwarding, or more complex setups.
If using SSL with the web application, then the developer also needs to generate and install the SSL cert and configure the web server correctly.

Vagrant

What is Vagrant? It is a set of tools to make it easier to use a virtual machine for your web development. It attempts to lessen many of the problems listed above through the use of automation. By design it also makes some assumptions about how you are using the VM. For example, it assumes that you have the source code for you project in a directory somewhere directly on your workstation and would prefer to use your favorite text editor on those files. Instead of expecting you to continually push updated files to your VM, it sets up a corresponding directory on the VM and keeps the two in sync for you (using either shared folders, NFS, Samba, or rsync). It also sets up the networking for accessing the VM, usually with port forwarding, so you don’t have to worry about that.

Benefits of Vagrant

Same as those listed above for using a VM, plus…
Flexible configuration (Vagrantfile) for creating and configuring the VM.
Automated networking for the VM with port forwarding. Abstracted ssh access (don’t need to set up a hostname for the VM, simply type vagrant ssh to connect). Port forwarded browser access to the VM (usually http://localhost:8080, but configurable).
Synced directory between your workstation and the VM for source code. Allows for developers to use their favorite text editor locally on their workstation without needing to manually copy files to the VM.
Expects the use of a configuration management system (like puppet, chef, salt, or bash scripts) to “provision” the VM (which could help with proper and consistent setup).
Through the use of Vagrant Cloud you can get a generated url for others to access your VM (makes it publicly available through a tunnel created with the command vagrant share).
Configuration (Vagrantfile and puppet/chef/salt/etc.) files can be maintained/reviewed by Operations engineers for consistency with production.

Problems with Vagrant

Still need to install and configure all services and packages. This is lessened with the use of a configuration management tool like puppet, but you still need to create/debug/maintain the puppet configuration and setup.
Backups of your work/configuration/everything are your own responsibility (extremely unlikely to happen). This may be lessened for VM configuration files, assuming they are included in your project’s VCS repo along with your source code.
Inexperienced developers can still break things, or change them to no longer match production (install arbitrary packages, different versions than what is in production, screw up the db, screw up Apache configuration, etc.).
If working with an established database, then downloading a dump, installing, and getting the database usable is time consuming and error prone. (“I just broke my dev database!” can be a complete blocker for development.)
If using SSL with the web application, then the developer also needs to generate and install the SSL cert and configure the web server correctly. This might be lessened if puppet (or whatever) is configured to manage this for you (but then you need to configure puppet to do that).

DevCamps

The DevCamps system takes a different approach. Instead of using VMs for development, it utilizes a shared server for all development. Each developer has their own account on the camps server and can create/update/delete “camps” (which are self-contained environments with all the parts needed). There is an initial setup for using camps which needs thorough understanding of the web application and all of its dependencies (OS, packages, services, etc.). For each camp, the system will create a directory for the user with everything related to that camp in it, including the web application source code, their own web server configuration, their own database with its own configuration, and any other resources. Each camp is assigned a camp number, and all services for that camp run on different ports (based on the camp number). For example, camp 12 may have Apache running on ports 9012 (HTTP) and 9112 (HTTPS) and MySQL running on port 8912. The developer doesn’t need to know these ports, as tools allow for easier access to the needed services (commands like mkcamp, re for restarting services, mysql_camp for access to the database, etc.).

DevCamps has been designed to address some of the pain usually associated with development environments. Developers usually do not need to install anything, since all dependencies should already be installed on the camps server (which should be maintained by an Operations engineer who can keep the packages, versions, etc. consistent with production). Having all development on a server allows Operations engineers to backup all dev work fairly easily. Databases do not need to be downloaded, manually setup, or anything- they should be set up initially with the camps system and then running mkcamp clones the database and sets it up for you. Running refresh-camp --db allows a developer to delete their camp’s database and get a fresh clone, ready to use.

Benefits of DevCamps

Each developer can create/delete camps as needed, allowing for multiple camps at once and multiple projects at once.
Operations engineers can manage/maintain all dependencies for development, ensuring everything is consistent with production.
Backups of all dev work is easy (Operations engineer just needs to backup the camps server).
Developer does not need to configure services (camp templating system auto-generates needed configuration for proper port numbers), such as Apache, nginx, unicorn, MySQL, Postgres, etc.
SSL certificates can be easily shared/generated/installed/etc. automatically with the mkcamp script. Dev environments can easily have HTTPS without the developer doing anything.
Developers should not have permissions to install/change system packages or services. Thus inexperienced developers should not be able to break the server, other developer’s environments, install arbitrary software. Screwing up their database or web server config can be fixed by either creating a new camp, refreshing their existing one, or an Operations engineer can easily fix it for them (since it is on a central server they would already have access to, and not need to worry about how to access some VM who knows where).

Problems with DevCamps

Since all camps live on a shared server running on different ports, this will not closely match production in that way. However, this may not be significant if nearly everything else does closely match production.
Adding a new dependency (for example, adding mongodb, or upgrading the version of Apache) may require quite a bit of effort and will affect all camps on the server- Operations engineer will need to install the needed packages and add/change the needed configuration to the camps system and templates.
Using your favorite text editor locally on your workstation doesn’t really work since all code lives on the server. It is possible to SFTP files back and forth, but this can be tedious and error prone.
Many aspects of the Operations (systems administration) side of the web application are hidden from the developer (this might also be considered a benefit).
All development is on a single server, which may be a single point of failure (if the camps server is down, then all development is blocked for all developers).
One camp can use up more CPU/RAM/disk/etc. then others and affect the server’s load, affecting the performance of all other camps.

Concluding Thoughts

It seems that Vagrant and DevCamps certainly have some good things going for them. I think it might be worth some thought and effort to try to meld the two together somehow, to take the benefits of both and reduce the problems as much as possible. Such a system might look like this:

Utilize vagrant commands and configuration, but have all VMs live on a central VM server. Thus allowing for central backups and access.
Source code and configuration lives on the server/VM but a synced directory is set up (sshfs mount point?) to allow for local editing of text files on the workstation.
VMs created should have restricted access, preventing developers from installing arbitrary packages, versions, screwing up the db, etc.
Configuration for services (database, web server, etc.) should be generated/managed by Operations engineers for consistency (utilizing puppet/chef/salt/etc.).
Databases should be cloned from a local copy on the VM server, thus avoiding the need to download anything and reducing setup effort.
SSL certs should be copied/generated locally on the VM server and installed as appropriate.
Sharing access to a VM should not depend on Vagrant Cloud, but instead should use some sort of internal service on the VM server to automate VM hostname/DNS for browser and ssh access to the VM.

I’m sure there are more pros and cons that I’ve missed. Add your thoughts to the comments below. Thanks.

Provisioning a Development Environment with Packer, Part 2

2014-03-14T00:00:00+00:00

In my previous post on provisioning a development environment with Packer I walked through getting a server setup with an operating system installed. This post will be focused setting up Ansible so that I can setup my development environment just the way I like it. Packer supports many different methods for provisioning. After playing with some of them, I decided that Ansible was a good mix of simplicity and functionality.

A Packer provisioner is simply a configuration template that is added to the json configuration file. The “provisioners” section of the configuration file takes an array of json objects which means that you aren’t stuck with just one kind of provisioner. For example, you could run some shell scripts using the shell provisioner, then upload some files using the File Uploads provisioner, followed by your devops tool of choice (puppet, salt, chef, or ansible). You can even roll-your-own provisioner if desired. Here’s an example provisioner setup for the shell provisioner:

{
  "variables": {...},
  "builders" : [...],
  "provisioners" [
    {
      "type": "shell",
      "inline": [ "echo foo" ]
    }
  ]
}

Sudo and User Considerations

Packer will login to the server over ssh and run your provisioners. The big headache that always comes out of this is some provisioners require sudo, or being logged in as root, to run their commands. Packer, however, will login as the user that was created during the build stage (see the “builders” section in the code snippet in the previous post). There are a couple of ways to handle this. First, you can do everything in packer as root. I don’t love this approach because I like to simulate the way that I setup a machine by hand and I never login as root if I can help it. The second method is to grant your user sudo access. This gets a little tricky so I’ll just show the a code snippet and then explain it below.

{
  "type": "shell",
  "execute_command": "echo '{{user `ssh_pass`}}' | {{ .Vars }} sudo -E -S sh '{{ .Path }}'",
  "inline": [
    "echo '%sudo    ALL=(ALL)  NOPASSWD:ALL' >> /etc/sudoers"
  ]
}

Utilizing the shell provisioner, the execute_command option is used to specify to the provisioner that whenever a command is run, use this command. The commands provided to the inline array are compiled into a single shell script which is injected as the .Path variable. To quote the Packer documentation:

The -S flag tells sudo to read the password from stdin, which in this case is being piped in with the value of [the user variable ssh_pass.] The -E flag tells sudo to preserve the environment, allowing our environmental variables to work within the script.

By setting the execute_command to this, your script(s) can run with root privileges without worrying about password prompts.

Taking advantage of this trick, a command can now be placed in the inline section that will add all members of the sudo group to the sudoers file granting them permission to use sudo without a password. Now I know this isn’t secure but for my purpose, which is to create a custom development enviornment on a Virtual Machine running only on my machine, this will be just fine. I also use this example to illustrate how to run commands as sudo.

The Ansible Provisioner

Once the shell provisioner is working, Ansible can be installed on the new machine and then executed using Packer’s Ansible provisioner. The easiest way to do this that I found was to have the shell provisioner install Ansible on the virtual machine as well as upload an ssh public key so that the Ansible user could log in. My “provisioners” section looks like this:

"provisioners": [
  {
    "type": "shell",
    "inline": [
      "mkdir .ssh",
      "echo '{{user `public_key`}}' >> .ssh/authorized_keys"
    ]
  },
  {
    "type": "shell",
    "execute_command": "echo '{{user `ssh_pass`}}' | {{ .Vars }} sudo -E -S sh '{{ .Path }}'",
    "inline": [
      "add-apt-repository ppa:rquillo/ansible",
      "apt-get update",
      "apt-get install -y ansible",
      "echo '%sudo    ALL=(ALL)  NOPASSWD:ALL' >> /etc/sudoers"
    ]
  },
  {
    "type": "ansible-local",
    "playbook_file": "site.yml"
  }
]

This configuration uses a variable in which I placed an ssh public key (I could also have used the File Uploads provisioner for this), installs Ansible from an updated PPA, and grants the user sudo priveliges via the sudo group as explained above. This also shows how you can execute more than one statement at a time using the “shell” provisioner.

Ansible Provisioning Tip

This tip could probably apply to any of the devops tools you’d like to use. If you are creating your Ansible yml files for the first time, you will likely run into the issue where you spend a lot of time waiting for the machine to build and provision only to discover your Ansible script is wrong. Troubleshooting becomes a problem because if anything fails during the provision, Packer will stop running and delete the Virtual Machine leaving you with no option other than fixing your mistake and then waiting for the entire process to run again.

One way I found around this is to take the last section of the provisioners array out, build your machine, and then move the base machine into a new directory once it’s been successfully built. From there you can start the machine manually and then run the ansible-playbook command from your local machine while you develop your playbook. Once you have a working playbook, add the ansible-local section back to the provisioners array and rebuild your machine with Packer. That should speed up your development and troubleshooting cycles.

A Hiccup with Ansible and Packer

Ansible allows you to create template files that can be used for configuration files and the like. According to the documentation, you can specify the location of the templates and other files using the playbook_paths option in the provisioner. I could not get this to work and after a lot of troubleshooting and looking at the code for the provisioner I am convinced there is a bug copying the playbook_paths directories to the remote machine. I’ve posted on the Packer discussion group about this but haven’t had any response on it yet. Once I get to the bottom the issue I’ll post an update here.

Conclusion

Packer has turned out to be a fabulous resource for me in quickly ramping up development environments. Ansible has also been a breath for fresh air for provisioning those machines. I’ve previously used chef and other devops tools which only led to a great deal of frustration. I’m happy to have some new tools in my belt that took very little time to learn and to get working.

Provisioning a Development Environment with Packer, Part 1

2014-03-12T00:00:00+00:00

I recently needed to reconstruct an old development environment for a project I worked on over a year ago. The codebase had aged a little and I needed old versions of just about everything from the OS and database to Ruby and Rails. My preferred method for creating a development environment is to setup a small virtual machine (VM) that mimics the production environment as closely as possible.

Introducing Packer

I have been hearing a lot of buzz lately about Packer and wanted to give it a shot for setting up my environment. Packer is a small command line tool written in the increasingly popular Go programming language. It serves three primary purposes:

Building a machine based on a set of configuration parameters
Running a provisioner to setup the machine with a desired set of software and settings
Performing any post processing instructions on that machine

Packer is really simple to install and I would refer you to their great documentation to get it set up. Once set up, you will have the packer command at your disposal. To build a new machine, all you need to is call:

packer build my_machine.json

The file my_machine.json can be the name of any json file and contains all the information Packer needs to setup your machine. The configuration json has three major sections: variables, builders, and provisioners. Variables are simply key value pairs that you can reference later in the builders and provisioners sections.

The Builder Configuration

Builders takes an array of JSON objects that specify different ways to build your machines. You can think of them as instructions on how to get your machine setup and running. For example, to get a machine up and running you need to create a machine, install an operating system (OS) and create a user so that you can login to the machine. There are many different types of builders, but for the example here, I’ll just use the vmware-iso machine type. Here’s a working JSON configuration file:

{
  "variables": {
    "ssh_name": "mikefarmer",
    "ssh_pass": "mikefarmer",
    "hostname": "packer-test"
  },

  "builders": [
    {
      "type": "vmware-iso",
      "iso_url": "os/ubuntu-12.04.4-server-amd64.iso",
      "iso_checksum": "e83adb9af4ec0a039e6a5c6e145a34de",
      "iso_checksum_type": "md5",
      "ssh_username": "{{user `ssh_name`}}",
      "ssh_password": "{{user `ssh_pass`}}",
      "ssh_wait_timeout": "20m",
      "http_directory" : "preseeds",
      "http_port_min" : 9001,
      "http_port_max" : 9001,
      "shutdown_command": "echo {{user `ssh_pass`}} | sudo -S shutdown -P now",
      "boot_command": [
        "<esc><esc><enter><wait>",
        "/install/vmlinuz noapic ",
        "preseed/url=http://{{ .HTTPIP }}:{{ .HTTPPort }}/precise_preseed.cfg ",
        "debian-installer=en_US auto locale=en_US kbd-chooser/method=us ",
        "hostname={{user `hostname`}} ",
        "fb=false debconf/frontend=noninteractive ",
        "keyboard-configuration/modelcode=SKIP keyboard-configuration/layout=USA ",
        "keyboard-configuration/variant=USA console-setup/ask_detect=false ",
        "initrd=/install/initrd.gz -- <enter>"
      ]
    }
  ]
}

The documentation for these settings is really good but I want to point out a few things that weren’t immediately clear. Some of these pertain mostly to the vmware-iso builder type, but I believe they are worth pointing out because some of them apply to other builder types as well.

First, the iso_url setting can be either an absolute path, a relative path, or a fully qualified url. The relative path is relative to the directory where you run the packer command. So here, when I run packer, I need to make sure that I do so from a directory that has an os subdirectory with the ubuntu iso located therein.

Next, once the ISO is downloaded, Packer will automatically start up your VMWare client and boot the virtual machine. Immediately after that, Packer will start up a VNC client and server along with a mini web server to provide information for your machine. The http_port_min and http_port_max specify which ports to use for the VNC clients. Setting them to the same will allocate just that port for it to use. The http_directory setting provides the name of a local directory to use for the mini web server as the document root. This is important for providing your VM with a preseed file, more about the preseed file will be discussed below.

Since we are using Ubuntu as our main machine, we will need to use sudo to send the shutdown command. The shutdown_command setting is used to gracefully shut down the machine at the conclusion of the run and provisioning of the machine.

Installing your OS

The boot_command is a series of keystrokes that you can send to the machine via VNC. If you have set up a Linux machine from scratch you know that you have to enter in a bunch of information to the machine about how to set it up for the first time such as time zone, keyboard layout, how to partition the hard drive, host name, etc. All these keystrokes needed to set up your machine can be used here.

But if you think about it, that’s a ton of keystrokes and this command could get quite long. A better way to approach this is to use a preseed file. A preseed.cfg file contains the same information you enter when you setup a machine for the first time. This isn’t something provided by Packer, but it is provided by the operating system to automatically provision machines. For Ubuntu, a preseed file is used like so:

When you boot from the startup media (in this case an ISO image), you can choose the location of the preseed file via a URL.
The preseed file is uploaded into memory and the configuration is read.
The installation process begins using information from the preseed file to enter the values where the user would normally enter them.

So how do we get the preseed file up to the machine? Remember that little web server that Packer sets up? Well, the IP address and port is made available to the virtual machine when it boots from the ISO. The following line tells the OS where to find the web server and the configuration file:

"preseed/url=http://{{ .HTTPIP }}:{{ .HTTPPort }}/precise_preseed.cfg"

Strings in Packer can be interpolated using a simple template format similar to mustache. The double curly braces tell Packer to insert a variable here instead of text. The HTTPIP and HTTPPort variables are made available by Packer to the template.

One more important note about the preseed file: You need to make sure that the settings for the username and password are the same as listed in your variables section so that you can login to the machine once it is built. Where do you get a preseed file? I found one on a blog titled Packer in 10 Minutes by @kappataumu. I only had to modify a few settings that were specific to my setup.

Remember that http_directory mentioned above? Well, that directory needs to include your preseed file. I’ve named mine precise_preseed.cfg for Ubuntu 12.04 Precise Pangolin.

Next up is provisioning but that is such a big topic by itself that I’ll move that into a separate blog post. The config file above will work as-is and once run it should setup a basic Ubuntu server for you. Go ahead and give it a try and let me know in the comments how it worked out for you.

Super Powers

I said that Packer has 3 primary purposes earlier. Well, I lied.

Packer’s superpower is that it can perform those 3 purposes over any number of machines, whether virtual, hosted, or otherwise in parallel. Supported machines are currently:

Amazon EC2 (AMI)
DigitalOcean
Docker
Google Compute Engine
OpenStack
QEMU
VirtualBox
VMware

Consider for a moment that you can now automatically setup and provision multiple machines with the same environment using a single command. Now you are seeing the power of Packer.

Mobile Emulation in Chrome DevTools

2014-01-24T00:00:00+00:00

I have been doing some mobile development lately and wanted to share the new Mobile Emulation feature in Chrome Canary with y’all. Chrome Canary is a development build of Chrome which gets updated daily and gives you the chance to use the latest and greatest features in Chrome. I’ve been using it as my primary browser for the past year or so and it’s been fairly stable. What’s great is that you can run Chrome Canary side-by-side with the stable release version of Chrome. For the odd time I do have issues with stability etc., I can just use the latest stable Chrome and be on my way. If you need more convincing, Paul Irish’s Chrome Canary for Developers post might be helpful.

I should mention that Chrome Canary is only available for OS X and Windows at this point. I tested Dev channel Chromium on Ubuntu 13.10 this afternoon and the new mobile emulation stuff is not ready there yet. It should not be long though.

Mobile Emulation in Chrome Dev Tools

Once enabled, the Emulation panel shows up in the Dev Tools console drawer. It gives you the option of emulating a variety devices (many are listed in the drop-down) and you also have the ability to fine tuning the settings à la carte. If you choose to emulate the touchscreen interface the mouse cursor will change and operate like a touch interface. Shift+drag allows you to pinch and zoom. There are some cool features for debugging and inspecting touch events as well.

Learning More

If you would like to learn more, be sure to check out the Mobile emulation documentation at the Chrome DevTools docs site.

Copy Data Between MySQL Databases with Sequel Pro

2014-01-10T00:00:00+00:00

Sequel Pro

I often use Sequel Pro when I’m getting up to speed on the data model for a project or when I just want to debug in a more visual way than with the mysql command-line client. It’s a free OS X application that lets you inspect and manage MySQL databases. I also find it very useful for making small changes to the data while I develop and test web apps.

Quickly Copy Data Between Databases

I recently needed a way to copy a few dozen records from one camp to another. I tried using the “SELECT…INTO OUTFILE” method but ran into a permissions issue with that approach. Using mysqldump was another option but that seemed like overkill in this case—I only needed to copy a few records from a single table. At this point I found a really neat and helpful feature in Sequel Pro: Copy as SQL INSERT

I simply selected the records I wanted to copy and used the “Copy as SQL INSERT” feature. The SQL insert statement I needed was now copied to the system clipboard and easily copied over to the other camp and imported via the mysql command-line client.

Bundles

The Sequel Pro website describes Bundles which extend the functionality in various ways—including copying data as JSON. Very handy stuff. Many thanks to the developers of this fine software. If you’re on OS X, be sure to give it a try.

Adventures with using Ruby 2.0 and libreadline

2013-05-17T00:00:00+00:00

I was asked to develop a prototype app for one of our clients lately. The basis for this app was an old Rails app:

Rails 3.2.8
RailsAdmin
MySQL
rbenv + ruby-build

I wanted to upgrade the stack to work with latest toys all cool kids are so thrilled about. I also didn’t have Rails console facility at my disposal since the Ruby version installed on the development machine hadn’t been compiled against libreadline.

Not having root or sudo access on the machine I embarked on a sligthly hacky journey to make myself a better working environment.

Ruby 2.0

After reading Mike Farmer’s blog post about Ruby 2.0 and tons of other material about it on the Internet, I wanted to get a feeling of how faster & greater the new Ruby is. It’s always great also to stay up-to-date with latest technologies. It’s great for me as a developer, and more importantly—it’s great for our clients.

Importance of libreadline in development with Ruby

To be productive developing any Rails-based application, we have to have Rails-console available at any moment. It serves a multitude of purposes. It’s also a great scratch-pad when developing methods.

While you don’t need your Ruby to support libreadline for basic uses of irb, you need it when using with Rails.

Installing Ruby 2.0.0 with rbenv (ruby-build)

If you’ve installed ruby-build some time ago, chances are that you need to update it in order to be able to install latest build of Ruby 2.0.0

To do it:

cd ~/.rbenv/plugins/ruby-build
git pull

And you should be able now to have available latest Ruby build to install:

rbenv install 2.0.0-p195

If you want to install Ruby compiled with support for libreadline, you have to have it installed in your system before compiling the build with rbenv install.

If you have access to root or sudo on your system, the easiest way is to e. g:

on Debian-related Linuxes:

apt-get install libreadline-dev

or on Fedora:

yum install readline-devel

Installing libreadline from sources

In my case—I had to download sources and compile them myself. Luckily the system had all needed essential packages installed for building it.

wget "ftp://ftp.cwru.edu/pub/bash/readline-6.2.tar.gz"
tar xvf readline-6.2.tar.gz
cd readline-6.2
./configure --prefix=/home/kamil/libs
make
make install

I had to specify –prefix option pointing at the path where I wanted the libreadline library to be installed after compilation.

Then, I was able to actually build Ruby with readline support “on”:

CONFIGURE_OPTS="--with-readline-dir=/home/kamil/libs" rbenv install 2.0.0-p195

Notice: I was making myself a development environment and compiling from sources was my last resort. It is not a good practice for production environments.

Last thing I needed to do was to get rb-readline working with the project I was working on.

It turnes out that latest rb-readline doesn’t play well with latest Ruby. Also, when using Ruby 2.0.0 one have to explicitely specify it in the Gemfile, or else it won’t be loaded for the console.

Gemfile:

gem 'rb-readline', '~> 0.4.2'

This still isn’t perfect

While this setup works, it won’t let you use arrow keys. The irb process crashes quickly after even first try to navigate through the text.

For some reason, after upgrading Ruby, the RailsAdmin stylesheets stopped working. I noticed that they are being served with comments which should be replaced by other stylesheets like:

/* ...
*= require_self
*= require_tree .
*/

I had to update Rails version in the Gemfile to have my admin back:

Gemfile:

gem 'rails', '3.2.13'

Console:

bundle

Last thing I wanted to do, was to try if I could upgrade Rails even further and have a working Rails4 setup. This was impossible unfortunately since RailsAdmin isn’t yet compatible with it as stated here.

I conclude that latest Ruby is quite usable right now. If you don’t mind the quirks with the readline—you’re pretty safe to upgrade. This assumes though that your app doesn’t use any incompatible elements.

The main Ruby site describes them like so:

There are five notable incompatibilities we know of:

The default encoding for ruby scripts is now UTF-8 [#6679]. Some people report that it affects existing programs, such as some benchmark programs becoming very slow [ruby-dev:46547].
Iconv was removed, which had already been deprecated when M17N was introduced in ruby 1.9. Use String#encode, etc. instead.
There is ABI breakage [ruby-core:48984]. We think that normal users can/should just reinstall extension libraries. You should be aware: DO NOT COPY .so OR .bundle FILES FROM 1.9.
#lines, #chars, #codepoints, #bytes now returns an Array instead of an Enumerator [#6670]. This change allows you to avoid the common idiom “lines.to_a”. Use #each_line, etc. to get an Enumerator.
Object#inspect does always return a string like #<ClassName:0x…> instead of delegating to #to_s. [#2152]
There are some comparatively small incompatibilities. [ruby-core:49119]

Test Web Sites with Internet Explorer for Free

2012-09-26T00:00:00+00:00

Browser Testing

While many Web Developers prefer to build sites and web applications with browsers like Chrome or Firefox it’s important for us to keep an eye on the browser market share for all web users. Internet Explorer (IE) still owns a large piece of this pie and because of this, it is important to test sites and applications to ensure they work properly when viewed in IE. This poses a potential problem for developers who do not use Windows.

Although I use OS X on my desktop, I have Windows virtual machines with IE 6,7,8,9 and 10 installed. I also have a Linux virtual machine running Ubuntu so I can check out Chrome/Chromium and Firefox on that platform. In the past I had tried solutions like MultipleIEs but wasn’t satisfied with them. In my experience I’ve found that the best way to see what users are seeing is to have a virtual machine running the same software they are.

I did some IE8 testing for a colleague a short time ago and suggested she should give VirtualBox a shot. Her response was “You should write a blog post about that!”. So here we are.

Free Tools

VirtualBox is a free virtualization application similar to Parallels or VMWare. These applications aim to solve this problem by allowing us to run different “guest” operating systems on the “host” operating system of our choice. For example, I run several versions of IE and Windows on my computer which is running OS X.

Microsoft have begun in recent years to release Virtual PC (VPC) images for free to enable Web Developers to test their sites and applications with various versions of Internet Explorer. The most recent images were released on July 30, 2012 and you can check them out here. Although these images are created to run on Microsoft’s Virtual PC application, with a little bit of effort we can also use them on Linux, OS X or Windows via VirtualBox.

For our purposes, we will be building a Windows 7 virtual machine running Internet Explorer 8. However, there are several images available with various Windows OS and Internet Explorer version combinations:

Windows 7 / IE8
Windows 7 / IE9
Windows Vista / IE7
Windows XP / IE6

There will likely be some differences creating virtual machines for the various images but the process will be similar to what we’ll document here for Windows 7 / IE.

Linux and OS X Users

In researching this article I came across a very helpful project on GitHub. The ievms project by xdissent is a bash script that automates this process. I was able to run the script and run my newly created Windows 7 / IE virtual machine shortly afterward.

Requirements

To start we’ll need to download and install VirtualBox. Be sure you have the most recent version available as it gets updated quite often. We’ll also need to install the VirtualBox Extension Pack. Both are available on the VirtualBox Downloads page.

Because the VPC images are quite large (~2.6GB in our case) they have been split up into several files. For the Windows 7 / IE8 image we’ll download the following files:

Once the files have been extracted we have created our virtual machine the disk will take up close to 10 GB. Make sure you have enough disk space available. I run my virtual machines on an external USB hard drive which works well if your local hard drive is starved for space.

Extract the VPC Files

Once you have all of the the VPC image files downloaded you’ll need to extract them:

Windows users: double-click the Windows_7_IE8.part01.exe file

Linux / OS X users: You will need to install unrar. I did so on OS X with Homebrew and there are packages available for Ubuntu as well.

Once you have unrar installed, issue the following command in the directory containing all of the VPC image files:

unrar e -y Windows_7_IE8.part01.exe

This will extract the files and combine them into a single .vhd file. The -y flag tells unrar to say “yes” to the EULA from Microsoft so be aware of that.

Create a new Machine

Once the .vhd file is ready we can create a new machine in VirtualBox. Start up VirtualBox and click the “New” button:

Click “Continue” to proceed.

VM Name and OS Type

We’ll enter a name (IE8 / Windows 7) and select “Microsoft” and “Windows 7” in the Operating System and Version drop-downs:

Click “Continue” to proceed.

Memory

Specify how much memory (RAM) you’d like to allocate to the virtual machine. This really depends on how much you have available but it is best to go with the minimum at the very least:

Click “Continue” to proceed.

Disk

Select the “Do not add a virtual hard drive” option.:

The reason we skip this for now is because we want to add the hard disk as an IDE drive. For some reason VirtualBox adds the disk as a SATA drive which causes the blue screen of death (BSOD) to appear when the virtual machine is booted up.

Click “Continue” to proceed and click “Continue” again when presented with the warning about not attaching a hard disk.

Review Settings and Create the VM

At this point you can review the settings you’ve made and click “Create” to complete the process:

Configure Storage Settings

Highlight your newly created virtual machine and click the “Settings” button. From there, click the “Storage” tab and add a hard disk to the IDE controller:

Absolute Pointing Device

In the Settings, click on the “System” tab and uncheck “Enable absolute pointing device”. I had to do this in order to to get the mouse working properly.

Start it Up

We’re almost there: next we need to start up the VM by clicking the “Start” button. You’ll see Administrator and IEUser accounts and the password for both is “Password1”. Log in as Administrator and check it out! It’s important to note when the virtual machine has the keyboard and mouse focus. When you click inside the virtual machine window, it will capture the input from your mouse and keyboard. To return focus back to your computer you will need to press the “host key” sequence on your computer:

The “host-key” sequence can be configured in the VirtualBox preferences. On my computer, the host-key sequence is the left CTRL key (displayed as Left ^). When the arrow icon is illuminated, the guest computer – Windows 7 in this case, has the focus. When I press the left CTRL key on my computer the arrow icon goes tray to indicate focus has been returned to my computer.

Activation

As soon as you log in you will be presented with the Windows Activation wizard. Follow the steps to activate this copy of Windows. Once that is complete you will be prompted to restart your computer. Go ahead and do that.

The screen resolution in the virtual machine will be quite low at first — I believe 800x600 is the default. Don’t worry about this as it will be addressed in the next step.

Guest Additions

Log in as Administrator (Password1) and Install the VirtualBox Guest Additions. To do this, navigate to Devices in the VirtualBox menu on your computer (the “host” machine"). Choose the menu option to “Install Guest Additions”. Each time you are prompted with the “Would you like to install…” dialog, choose “Install”:

This will install some drivers optimized by VirtualBox which allow you to change the screen resolution, improve the interaction between your host computer and the Windows guest operating system. Once the install process is complete you will be prompted to restart your virtual machine once more.

Finished Product

With Windows activated and the Guest Additions installed you should be able to log in to your very own Windows 7 / IE8 testing machine!

This free copy of Windows will operate for 30 days and the trial period can be extended twice by running the following at the command prompt:

slmgr –rearm

Check out the documentation at Microsoft for more details.

GNU Screen + SSH_AUTH_SOCK; my new approach

2011-02-23T00:00:00+00:00

Over the years, I’ve played around with several different methods of keeping my forwarded SSH-Agent authentication socket up-to-date in long-running screen sessions (referenced via the $SSH_AUTH_SOCK shell variable). The basic issue here is that Screen sees the process environment at the time it was initially launched, not that which exists when reattaching in a subsequent login session. This means that the $SSH_AUTH_SOCK variable as screen sees it will refer to a socket which no longer exists (as it was removed when you logged out after detaching on the initial login when starting screen).

Some of my previous methods have included a hard-coded variable for the socket itself (downsides: if it’s a predictable name you’re potentially opening some security issues, plus if you open multiple sessions to the same account, you kill the latest socket), symlinking the latest $SSH_AUTH_SOCK to a hard-coded value on login (similar issues), dumping $SSH_AUTH_SOCK to a file, and aliasing ssh and scp to first source said file to populate the local window’s enviroment (doesn’t work in scripts, too much manual setup when adapting to a new system/environment, won’t work with any other subsystem not already explicitly handled, etc).

Recently though, I’ve come up with a simple approach using screen’s -X option to execute a screen command outside of screen and just added the following to my .bashrc:

screen -X setenv SSH_AUTH_SOCK "$SSH_AUTH_SOCK"

While not perfect, in my opinion this is a bit of an improvement for the following reasons:

It’s dirt-simple. No complicated scripts to adjust/maintain, just a command that’s almost completely self-explanatory.
It doesn’t kill the environment for existing screen windows, just adjusts the $SSH_AUTH_SOCK variable for new screen windows. This ends up matching my workflow almost every time, as unless a connection dies, I leave the screen window open indeterminately.
If you have multiple sessions open to the same account (even if not running both in screen), you’re not stomping on your existing socket.
Did I mention it’s dirt-simple?

There are presumably a number of other environment variables that would be useful to propagate in this way. Any suggestions or alternate takes on this issue?

Managing Perl environments with perlbrew

2011-02-21T00:00:00+00:00

As a Perl hobbyist, I’ve gotten used to the methodical evolution of Perl 5 over the years. Perl has always been a reliable language, not without its faults, but with a high level of flexibility in syntactical expression and even deployment options. Even neophytes quickly learn how to install their own Perl distribution and CPAN libraries in $HOME. But the process can become unwieldy, particularly if you want to test across a variety of Perl versions.

To contrast, Ruby core development frequently experiences ABI breakages, even between minor releases. In spite of the wide adoption of Ruby as a Web development language (thanks to Ruby on Rails), Ruby developers are able to plod along unconcerned, where these incompatibilities would almost certainly lead to major bickering within the Perl or PHP communities. How do they do it? The Ruby Version Manager.

Ruby Version Manager (RVM) allows users to install Ruby and RubyGems within their own self-contained environment. This allows each user to install all (or only) the software that their particular application requires. Particularly for Ruby developers, this provides them with the flexibility to quickly test upgrades for regressions, ABI changes and enhancements without impacting system-wide stability. Thankfully a lot of the ideas in RVM have made their way over to the Perl landscape, in the form of perlbrew.

Perlbrew offers many of the same features found in RVM for Ruby. It’s easy to install. It isolates different Perl versions and CPAN installations in your $HOME and helps you switch between them. It automates your environment setup and teardown. And most importantly, using perlbrew means not having to clutter your default system Perl with application-specific CPAN dependencies.

Getting started with perlbrew couldn’t be easier. A quick one-liner is all it takes to install perlbrew in your home directory.

$ curl -L http://xrl.us/perlbrewinstall | bash

If you need to install perlbrew somewhere other than your home directory, just download the installer and pass it the PERLBREW_ROOT environment variable.

$ curl -LO http://xrl.us/perlbrew
$ chmod +x perlbrew
$ PERLBREW_ROOT=/mnt/perlbrew ./perlbrew install

Follow the instructions on screen and you’ll be ready to use perlbrew in no time. The perlbrew binary will be installed in ~/perl5/perlbrew/bin, so make sure to adjust your login $PATH accordingly.

Once you’re done installing perlbrew there are a couple commands you’ll want to run before installing your own Perl versions or CPAN modules. The perlbrew init command is mandatory; this initializes your perlbrew directory. It can also be used later if you need to modify your PERLBREW_ROOT setting. The perlbrew mirror is optional (but recommended) to help you select a preferred CPAN mirror.

$ perlbrew init
$ perlbrew mirror

Next comes the fun part. Start off by verifying the Perl version(s) that perlbrew sees.

$ perlbrew list
* /usr/bin/perl (5.10.1)

Install a newer version of Perl.

$ perlbrew install 5.12.3

Now switch to the newer Perl.

$ perlbrew list
* /usr/bin/perl (5.10.1)
perl-5.12.3

$ perlbrew switch perl-5.12.3

$ perlbrew list
/usr/bin/perl (5.10.1)
* perl-5.12.3

$ perl -v

This is perl 5, version 12, subversion 3 (v5.12.3) built for x86_64-linux

Copyright 1987-2010, Larry Wall

Perl may be copied only under the terms of either the Artistic License or the
GNU General Public License, which may be found in the Perl 5 source kit.

Complete documentation for Perl, including FAQ lists, should be found on
this system using “man perl” or “perldoc perl”. If you have access to the
Internet, point your browser at http://www.perl.org/, the Perl Home Page.

Alternatively, if you only want to test a different Perl version, try the perlbrew use command (note: this only works in bash and zsh). Unlike the switch command, use is only active for the current shell.

$ perlbrew use system

$ perlbrew list
* /usr/bin/perl (5.10.1)
perl-5.12.3

A quick peek behind the curtain reveals much of the simplicity behind perlbrew.

$ ls -l ~/perl5/perlbrew/
total 2680
-rw-r--r--  1 testy  users      408 Feb 10 23:58 Conf.pm
drwxr-xr-x  2 testy  users      512 Feb 10 23:46 bin
drwxr-xr-x  4 testy  users      512 Feb 11 09:59 build
-rw-r--r--  1 testy  users  1333196 Feb 11 10:33 build.log
drwxr-xr-x  2 testy  users      512 Feb 11 09:59 dists
drwxr-xr-x  2 testy  users      512 Feb 10 23:47 etc
drwxr-xr-x  4 testy  users      512 Feb 11 10:32 perls

$ ls -l ~/perl5/perlbrew/perls/
total 8
drwxr-xr-x  5 testy  users  512 Feb 11 00:38 perl-5.12.3
drwxr-xr-x  5 testy  users  512 Feb 11 10:32 perl-5.13.6

If you’re a Perl developer, the perlbrew project may help alleviate a lot of the pain associated with team development or multi-tenant programming environments. Suddenly it becomes much easier to manage your own software requirements, resulting in faster development and testing cycles for you, and fewer headaches for your System Administrators.

Visit at DistribuTECH

2011-02-02T00:00:00+00:00

I had the chance to attend DistribuTECH in San Diego, CA this past week. DistribuTECH is billed as the utility industry’s leader in smart grid conference and exposition. End Point was present at the conference on behalf of Silver Spring Networks. Silver Spring Networks contracted with us to provide a Liquid Galaxy installation for their exhibit.

The Liquid Galaxy did its job from what I could tell. The exhibit was consistently surrounded with conference goers both interested in listening and watching the tours that were being presented as well as wanting to see what the Liquid Galaxy was all about. This was the first time I had seen the Liquid Galaxy and was quite impressed with how well it worked. I saw many people moving their bodies in sync with what was being displayed on the screen, showing that they felt immersed while within the galaxy. One gentleman knelt down while attempting to look under a graph that was being presented on the screen. This same person had returned to the exhibit several times, bringing colleagues back each time to “show off” what he had found.

I spent some time on the conference floor, checking out what was being displayed and seeing how others were getting the attention of the attendees. I could not find anything that compared to the Liquid Galaxy in both wow factor and usability. The fine folks at Silver Spring Networks also seemed impressed with the reaction they were receiving.

One product that I found interesting while walking the floor was a large unit that would freeze itself at night, when power is less expensive, the draw on the grid is less, and it is cooler outside and then use the ice as a coolant to be pumped into the AC unit during the day resulting in a reduction of cost and energy usage to cool a building.

I also took a look at the Silver Spring Networks products on display. One that interested me was their home portal that allowed customers that had Silver Spring smart meters installed by their utility to visit a web portal and view several things including their current usage and compare their usage to that of their neighbors. I can see how someone concerned with the environment could use this information to lessen their power usage.

Keep an eye out here for a few blog posts from Adam on his experience with tour development for the Liquid Galaxy.

Debugging jQuery

2010-09-28T00:00:00+00:00

A recent reskin project for a client requires that we convert some of their old Prototype code to jQuery as well as create new jQuery code. I have not done much with converting Prototype to jQuery and I felt like my debugging tools for JavaScript were under par. So this morning I set out to find what was available for jQuery and I found this article on the subject.

I’ve used Firebug for some time now, but was unaware of some of the supporting plugins that would certainly help with debugging JavaScript. Some of the plugins and methods found in the article that I found immediately helpful were:

FireFinder: Makes it quite easy to verify that the selector values in your code are correct and that the proper elements are returned. I was able to immediately pinpoint problems with my selectors and this brought to light why certain events were not firing.
Firebug Console: Using the console.log function allowed me to check values without littering my code with alert statements.
FireQuery: At a glance this plugin for Firebug shows which elements have event handlers bound to them.
Firebug Breakpoints: Setting breakpoints and watch statements in your code makes it easier to see what is happening in the JavaScript code as it is executed instead of trying to figure out what happened after the code has run its course.

Thanks to the author of the article, Elijah Manor, for the in-depth information on debugging jQuery code.

Guidelines for Interchange site migrations

2010-09-03T00:00:00+00:00

I’m involved at End Point often with Interchange site migrations. These migrations can be due to a new client coming to us and needing hosting or migrating from one server to another within our own infrastructure.

There are many different ways to do a migration, in the end though we need to hit on certain points to make sure that the migration goes smoothly. Below you will find steps which you can adapt for your specific migration.

During the start of the migration it might be a good time to introduce git for source control. You can do this by creating the repository and cloning it to /home/account/live, setting up .gitignore files for logs, counter files, gdbm files. Then commit the changes back to the repo and you’ve now introduced source control without much effort, improving the ability to make changes to the site in the future. This is also helpful to document the changes you make to the code base along the way during the migration in case you need to merge changes from the current production site before completing the migration.

Export all of the gdbm databases to their text file equivalents on the production server
Take a backup from production of the database, catalog, interchange server, htdocs
Setup an account
Create the database and user
Restore the database, catalog, interchange server and htdocs
Update the paths in interchange/bin for each script to point at the new location
Grep the restored code for hard coded paths and update those paths to the new locations. Better yet move these paths out to a catalog_local.cfg where environment specific information can go.
Grep the restored code for hard coded urls and use the [area] tag to generate the urls
Update the urls in products/variable.txt to point at the test domain
Update the sql settings in products/variable.txt to point at the new database using the new user
Remove the gdbm databases so they will be recreated on startup from the source text files
Install a local Perl if it’s not already installed (./configure -des will compile and install Perl locally)
Install Bundle::InterchangeKitchenSink
Install the DBD module for MySQL or PostgreSQL
Review the code base looking for use statements in custom code and Require module settings in interchange.cfg. Install the Perl modules found into the local Perl.
Setup a non ssl and ssl virtual host using a temporary domain. Configure the temporary domain to use the SSL certificate from the production domain.
Firewall or password protect the virtual host so it is not accessible to the public
Generate a vlink using interchange/bin/compile and copy it into the cgi-bin directory and name it properly
Startup the new Interchange
Review error messages and resolve until Interchange will start properly
Test the site thoroughly, resolving issues as they appear. Make sure that checkout, charging credit cards, sending of emails, using the admin, etc all function.
Migrate any cron jobs running on the current production site, such as session expiration scripts
Setup logrotation for the new logs that will be created
Verify that you have access to make DNS changes
Set the TTL for the domain to a low value such as 5 minutes
Modify the new production site to respond to the production url, test by updating your hosts file to manually set the IP address of the domain
Shutdown the new Interchange
Restore a copy of the original backup for Interchange, the catalog and htdocs to /tmp on the production server
Shutdown the production Interchange, put up a maintenance note on the production site.
Take a backup of the production database and restore on the new server
Diff the Interchange, catalog and htdocs directory between /tmp and the current production locations, making note of the files that have changed since we took the original copy.
Copy the files that have changed, making sure to merge with any changes we have made on the new production site. Making sure to copy over all .counter and .autonumber files to the new production site.
Start Interchange on the new production server
Test the site thoroughly on the new production server, using the production url. Make sure that checkout with charging the credit card functions properly.
Resolve any remaining issues found during the testing
Setup the Interchange daemon to start at boot for this site in /etc/rc.d/rc.local or in cron using @reboot
Update DNS to point at the new production IP address
Update the TTL of the domain to a longer value
Open the site to the public by opening the firewall or removing the password protection
Keep an eye on the error logs for any issues that might crop up

This will hopefully give you a solid guide for performing an Interchange site migration from one server to another and some of the things to watch out for that might cause issues during the migrations.

Why is my load average so high?

2010-07-15T00:00:00+00:00

One of the most common ways people notice there’s a problem with their server is when Nagios, or some other monitoring tool, starts complaining about a high load average. Unfortunately this complaint carries with it very little information about what might be causing the problem. But there are ways around that. On Linux, where I spend most of my time, the load average represents the average number of process in either the “run” or “uninterruptible sleep” states. This code snippet will display all such processes, including their process ID and parent process ID, current state, and the process command line:

#!/bin/sh

ps -eo pid,ppid,state,cmd |\
    awk '$3 ~ /[RD]/ { print $0 }'

Most of the time, this script has simply confirmed what I already anticipated, such as, “PostgreSQL is trying to service 20 times as many simultaneous queries as normal.” On occasion, however, it’s very useful, such as when it points out that a backup job is running far longer than normal, or when it finds lots of “[pdflush]” operations in process, indicating that the system was working overtime to write dirty pages to disk. I hope it can be similarly useful to others.

dstat: better system resource monitoring

2009-12-19T00:00:00+00:00

I recently came across a useful tool I hadn’t heard of before: dstat, by Dag Wieers (of DAG RPM-building fame). He describes it as “a versatile replacement for vmstat, iostat, netstat, nfsstat and ifstat.”

The most immediate benefit I found is the collation of system resource monitoring output at each point in time, removing the need to look at output from multiple monitors. The coloring helps readability too:

% dstat

—-total-cpu-usage—- -dsk/total- -net/total- —paging– —system–

usr sys idl wai hiq siq| read writ| recv send| in out | int csw

4 1 92 3 0 0| 56k 84k| 0 0 | 94B 188B|1264 1369

3 7 43 44 1 1| 368k 11M| 151B 222B| 0 260k|1453 1565

3 2 46 48 1 0| 432k 5784k| 0 0 | 0 0 |1421 1584

2 2 47 49 0 0| 592k 0 | 0 0 | 0 0 |1513 1763

6 2 44 49 1 0| 448k 248k| 0 0 | 0 0 |1398 1640

8 4 41 45 3 0| 456k 0 | 135B 222B| 0 0 |1530 2102

18 4 38 41 0 0| 408k 128k| 0 47B| 0 0 |1261 1977

10 4 44 43 0 0| 728k 208k| 0 0 | 0 0 |1445 2203

6 3 39 51 0 0| 648k 256k|3607B 4124B| 0 0 |1496 2180

7 7 34 53 0 0|1088k 0 |1234B 582B| 0 0 |1465 2057

14 8 28 49 0 0|2856k 104k| 0 0 | 0 52k|1610 2995

6 6 43 45 0 0|1992k 0 |5964B 4836B| 0 0 |1493 2391

9 14 34 44 0 0|2432k 112k|7854B 726B| 0 0 |1527 2190

9 11 40 41 1 0|2680k 0 |1382B 972B| 0 0 |1550 2298

5 4 68 22 0 0| 576k 1096k| 12k 4628B| 0 0 |1522 1731 ^C

(Textual screenshot by script of util-linux and Perl module HTML::FromANSI.)

Its default one-line-per-timeslice output makes it good for collecting data samples over time, as opposed to full-screen top-like utilities such as atop, which give much more detailed information at each snapshot, but don’t show history.

Since dstat is a standard package available in RHEL/CentOS and Debian/Ubuntu, it is a reasonably easy add-on to get on various systems.

dstat also allows plugins, and just in the most recent release last month were added new plugins “for showing NTP time, power usage, fan speed, remaining battery time, memcache hits and misses, process count, top process total and average latency, top process total and average CPU timeslice, and per disk utilization rates.”

It sounds like it’ll grow even more useful over time and is worth keeping an eye on.

GNU Screen: follow the leader

2009-09-24T00:00:00+00:00

First of all, if you’re not using GNU Screen, start now :).

Years ago, Jon and I spoke of submitting patches to implement some form of “follow the leader” (like the children’s game, but with a work-specific purpose) in GNU Screen. This was around the time he was patching screen to raise the hard-coded limit of 40 windows allowed within a given session, which might give an idea of how much screen gets used around here (a lot).

The basic idea was that sometimes we just want to “watch” a co-worker’s process as they’re working on something within a shared screen session. Of course, they’re going to be switching between screen windows and if they forget to announce “I’ve switched to screen 13!” on the phone, then one might quickly become lost. What if the cooperative work session doesn’t include a phone call at all?

To the rescue, Screen within Screen.

Accidentally arriving at one screen session within another screen session is a pretty common “problem” for new screen users. However, creative use of two (or more) levels of nested screen during a shared session allows for a “poor man’s” follow the leader.

If the escape sequence of the outermost screen is changed to something other than the default, then the default escape sequence will pass through and take effect on the inner screen. In this way, anyone attached to the outermost screen will be following whomever is controlling the inner screen session as they flip between windows, grep logs, launch editors and save my vegan bacon! To “break away” from the co-working session, a user would simply use the chosen non-default escape sequence of the outermost screen to create a new window or disconnect entirely.

Sound confusing? Give some of the following commands a try. You can always just close out all the windows of a screen session and eventually you’ll make it back to your original shell.

Steps:

start the outermost screen session (called “followme”) with a non-default escape sequence (pick one that suits you):

screen -S followme -e ^ee

from within the “followme” session, start the inner screen where actual work will be performed:

screen -S work

get friends and co-workers (logged-in as the same user) to connect to your “followme” screen:

screen -x followme

work as normal using the default: CTRL-a sequences (which ought to affect the inner “work” session).
to “break away” from the “work” session: CTRL-e sequences (which ought to affect the outer “followme” session). For example, to disconnect from the shared session, one would type CTRL-e d

Note: If those sharing the screen session are already acclimated to screen-within-screen, you can skip the non-default escape sequences entirely and use CTRL-a a as the escape sequence (another a for every level of screen-within-screen). This also happens to be your evasion route for accidental screen-within-screen moments.

Remember that, by default, everyone who wants to share the screen must already be logged-in as the same user (without the use of sudo or su). There are methods of allowing shared screen access between users, but those are outside the scope of this post.

Have fun!

(Update: Note that tmux by default has every attached viewer follow whoever is driving, and there you have to do extra work to have separate control of windows. tmux was still new and not available in most production systems at the time this article was written.)

MTU tweak: a fix for upload pain

2009-07-11T00:00:00+00:00

While traveling and staying at Hostel Tyn in Prague’s city center, I ran into a strange problem with my laptop on their wireless network.

When many people were using the network (either on the hostel’s public computers or on the wireless network), sometimes things bogged down a bit. That wasn’t a big deal and required merely a little patience.

But after a while I noticed that absolutely no “uploads” worked. Not via ssh, not via browser POST, nothing. They always hung. Even when only a file upload of 10 KB or so was involved. So I started to wonder what was going on.

As I considered trying some kind of rate limiting via iptables, I remembered somewhere hearing that occasionally you can run into mismatched MTU settings between the Ethernet LAN you’re on and your operating system’s network settings.

I checked my setup and saw something like this:

ifconfig wlan0
wlan0     Link encap:Ethernet  HWaddr xx:xx:xx:xx:xx:xx
          inet addr:10.x.x.x  Bcast:10.x.x.x  Mask:255.255.255.0
          inet6 addr: fe80::xxx:xxxx:xxxx:xxxx/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1239 errors:0 dropped:0 overruns:0 frame:0
          TX packets:20 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:191529 (191.5 KB)  TX bytes:4543 (4.5 KB)

The MTU 1500 stood out as being worthy of tweaking. So I tried a completely unscientific change:

sudo ifconfig wlan0 mtu 1400

Then tried the same HTTP POST that had been consistently failing, and poof! It worked fine. Every time.

I think mostly likely something more than 1400 bytes would’ve been possible, perhaps just a few short of 1500. The number 1492 rings familiar. I’ll be old-fashioned and not look it up on the web. But this 1400-byte MTU worked fine and solved the problem. To my delight.

As an interesting aside, before making the change, I found one web application where uploads did work fine anyway: Google’s Picasa. I’m not sure why, but maybe it sliced & diced the upload stream into smaller chunks on its own? A mystery for another day.

Operating system upgrades

2009-05-12T00:00:00+00:00

This won’t be earth-shattering news to anyone, I hope, but I’m pleased to report that two recent operating system upgrades went very well.

I upgraded a laptop from Ubuntu 8.10 to 9.04, and it’s the smoothest I’ve ever had the process go. The only problem of any kind was that the package download process stalled on the last of 1700+ files downloaded, and I had to restart the upgrade, but all the cached files were still there and on reboot everything worked including my two-monitor setup, goofy laptop audio chipset, wireless networking, crypto filesystem, and everything else.

I also upgraded an OpenBSD 4.3 server that is a firewall, NAT router, DHCP server, and DNS server, to OpenBSD 4.5. It was the first time I used the in-place upgrade with no special boot media and fetching packages over the network, as per the bsd.rd instructions, and it went fine. Then the extra packages that were there before had to be upgraded separately as per the FAQ on pkg updates. I initially scripted some munging of pkg_info’s output, not realizing I could simply run pkg_add -u and it updates all packages.

There was one hangup upgrading zsh, which I just removed and reinstalled. Everything else went fine, and all services worked fine after reboot.

How pleasant.

TLS Server Name Indication

2009-05-08T00:00:00+00:00

I came across a few discussions of the TLS extension “Server Name Indication”, which would allow hosting more than one “secure” (https) website per IP address/TCP port combination. The best summary of the state of things is (surprise, surprise) the Wikipedia Server Name Indication article. There are more details about client and server software support for SNI in Zachary Schneider’s blog post and Daniel Lange’s blog post.

I don’t recall hearing about this before, but if I did I probably dismissed as being irrelevant at the time because there would’ve been almost no support in either clients or servers. But now that all major browsers on all operating systems support SNI except some on Windows XP it may be worth keeping an eye on this.

Yes, IE on Windows XP is still a huge contingent and thus a huge hurdle. But maybe Microsoft will backport SNI support to XP. Even if just for IE 7 and later. Or maybe we’ll have to wait a few more years till the next Windows operating system (hopefully) displaces XP. Here’s a case where the low popularity of Vista (which supports SNI) is hurting the rest of us.

I’m really looking forward to the flexibility of name-based virtual hosting for https that we’ve had for 10+ years with plain http. It could really change the setup and ongoing infrastructure costs for secure websites, such as ecommerce sites.