Data Center Relocation Initiatives: Daunting But Achievable

2018-12-20T00:00:00+00:00

Introduction

A customer asked for our help dealing with logistical nightmares they encountered during a hardware update and data center relocation project. The customer had two active data centers, and wanted to relocate one of them to a modern Tier 4 facility to improve its performance and provide redundancy for critical systems. They also wanted to consolidate or decommission equipment to reduce recurring expenses and reduce their carbon footprint. The client decided to move to a Tier 4 data center because they provide across-the-board redundancy within the data center.

Moving forward with Tier IV

Enterprises typically transition to Tier 4 data centers because they offer the highest uptime guarantees, and have no single points of failure. These facilities are fully redundant in terms of electrical circuits, cooling, and networking. This architecture is best able to withstand even the most serious technical incidents without server availability being affected. Tier IV facilities have contracts with disaster management companies who will provide them, for example, with fuel in the event that a natural disaster damages the power grid.

(More information about the four data center levels is at the end of this post.)

Key Issues

Some of the customer’s key concerns were:

1. To safely move their virtual environment to the new data center.

2. To protect assets during the relocation.

3. To completely shut down the current data center and migrate seamlessly to the new data center.

4. To update external DNS records without causing any downtime for the web applications used by their customers.

Project Planning

To manage data center relocations is challenging because there are many moving parts and variables. Typically, planning starts 6 months ahead of the scheduled relocation date. We seek to understand the role of each system by consulting in-depth with all the teams involved. We analyze each part of the client’s tech infrastructure, break it down, and strategize about the best approach for handling the migration of each component. Additional time to plan can help especially when a large group of stakeholders is involved. Successful planning requires hard and sometimes extensive work, but makes an otherwise overwhelming data center migration smooth and worry free.

Here are some of our procedures for pre-migration preparation:

1. Process the paperwork for new hardware. Identify, review, and approve all applications. Hire movers. Schedule and coordinate with internal management and key stakeholders like IT staff, system admins, contractors, circuit providers, etc.

2. Collect and analyze resource usage associated with the critical and non-critical systems and compare resource needs to resource availability at the new facility.

3. Design and schedule the migration of critical systems to the new data center. Discuss the plan with key stakeholders.

4. Make sure replicas of critical systems are up to date.

5. Set up a new environment with new hardware in the new data center and establish network connections.

6. Prepare hardware and virtualization technology in the new data center.

7. Test, retest, and burn in the new hardware (2–4 weeks of testing).

8. Create a non-essential server within the virtualization environment and test it.

A Challenge: Eliminating Downtime

To minimize or eliminate downtime requires very careful preparation. With this particular client, we had the luxury of their already having a secondary data center, which could accommodate a majority of the critical systems during the migration.

We originally planned to leverage this by simply changing the external DNS records to point to this data center during the migration, but during planning determined that the external DNS record would not replicate worldwide quickly enough to meet the uptime needs of their global operation.

Solutions

After discussing various approaches, we ended up using a load balancer. A load balancer is a network device that routes incoming traffic destined for a single destination (web site, application, or service) and ‘shares’ the incoming connections across multiple destination devices or services. In this case, the data center we were to migrate to housed the client’s critical systems like web servers, database, etc. The secondary data center housed a replica of the critical systems using a replication technology, e.g. Actifio, Rubrik, SAN mirroring, VMWare SRM, or Zerto — we used Actifio in this scenario.

The customer had multiple systems with external DNS records and public IP addresses. We did not change the public DNS names for the critical systems. We did change the public IP addresses associated with the public DNS names, to point to the load balancer, which then pointed to the appropriate data center throughout the process. This setup enabled a gradual, flexible, and fully controlled transition to the new data center.

Summary

Our client wanted to reduce their servers’ power usage 40% by consolidating into a hyper-converged system. Other goals were to reduce floor space by 50%, and monthly recurring data center charges by 50%. We transitioned them to a new hardware stack (compute, storage, network, backup environment) in a Tier 4 data center, and saved them 50% percent on data center operation costs. Additionally, during the process we corrected issues associated with disaster recovery and business continuity, leaving the client’s operations more resilient, affordable, and more secure.

Appendix: Data Center Classifications

Tier IV (Best)

Tier IV data centers have redundancies for every process and data protection stream. No single outage or error can shut down the system. To keep Tier IV ranking, data centers must have 99.995% minimum uptime per year. They come equipped with 2N+1 infrastructure (two times the amount required for operation plus a backup), which is considered “fully redundant.”

Tier III (Average)

N+1 (the amount required for operation plus a backup) fault tolerance, which means a Tier III facility can undergo routine maintenance without a hiccup in operations. 99.982% minimum uptime per annum, used for maintenance and overwhelming emergencies.

Tier II (Minimal)

99.749% minimum uptime. Tier II facilities have considerably more downtime than Tier III facilities, primarily because Tier II facilities have less redundancy. They do however have partial cooling redundancy and multiple power redundancies.

Tier I (Basic, Cheapest)

99.671% minimum uptime, with 28.8 hours of downtime each year.

Disaster Recovery — Miami to Dallas in one hit

2017-10-03T00:00:00+00:00

Hurricane Irma came a knocking but didn’t blow Miami away.

Hurricanes are never fun, and neither is knowing that your business servers may be in the direct path of a major outage due to a natural disaster.

Recently we helped a client prepare a disaster recovery environment, 4 days out pending Hurricane Irma approaching Miami—where it happens their entire server ecosystem sat. In recent months we had been discussing a long-term disaster recovery plan for this client’s infrastructure, but the final details hadn’t been worked out yet, and no work begun on it, when the news of the impending storm started to arrive.

Although the Miami datacenter they are using is highly regarded and well-rated, the client had the foresight to think about an emergency project to replicate their entire server ecosystem out of Miami to somewhere a little safer.

Fire suits on, battle helmets ready, GO! Two team members jumped in and did an initial review of the ecosystem: six Linux servers, one Microsoft SQL Server database with about 50 GB of data, and several little minions. All of this hosted on a KVM virtualization platform. OK, easy, right? Export the VM disks and stand up in a new datacentre on a similar KVM-based host.

But no downtime could be scheduled at that time, and cloning the database would take too much time to make a replica. If we can’t clone the VM disks without downtime we need another plan. Enter our save-the-day idea: use the database clone from the development machine.

So first things first: Build the servers in the new environment using the cloned copy, rsync the entire disk across, hoping that a restart would return the machine into a working state, take the development database and use the Microsoft tools at our disposal to replicate one current snapshot, then replicate row level details, in preparation of reducing the amount of lost data, should the datacenter lose connection.

Over the course of the next 16 hours, with 2 DevOps engineers, we replicated 6 servers, migrated 100+ GB of data twice and had a complete environment ready for a manual cutover.

Luckily the hurricane didn’t cause any damage to the datacenter, which never lost power or connectivity, so the sites never went down.

Even though it wasn’t used in this case, this emergency work did have a silver lining: Until this client’s longer-term disaster recovery environment is built, these system replicas can serve as cold standby, which is much better than having to rebuild from backups if their primary infrastructure goes out.

That basic phase is out of the way, so the client can breathe easier. Along the way, we produced some updated documentation, and gained deeper knowledge of the client’s software ecosystem. That’s a win!

The key takeaways here: Don’t panic, and do a little planning prior. Leaving things to the last minute rarely works well. And preferable to all of the above: Plan and build your long-term disaster recovery environment now, well before any possible disaster is on the horizon!

To all the people affected by Hurricane Irma we wish a speedy rebuild.

We are bigger than VR gear - Liquid Galaxy

2016-04-27T00:00:00+00:00

Nowadays, virtual reality is one of the hottest topics in tech, with VR enabling users to enter immersive environments built up by computer technology. I attended Mobile World Congress 2016 a few weeks ago, and it was interesting to see people sit next to one another and totally ignore one another while they were individually immersed in their own virtual reality worlds.

When everyone is so addicted to their little magic boxes, they tend to lose their connections with people around them. End Point has developed a new experience in which users can watch and share their virtually immersive world together. This experience is called the Liquid Galaxy.

When a user stands in front of Liquid Galaxy and is surrounded by a multitude of huge screens arranged in a semicircle, he puts not only his eyes but his whole body into an unprecedented 3D space. These screens are big enough to cover the audience’s entire peripheral vision and bring great visual stimulation from all directions. When using the Liquid Galaxy system, the users become fully immersed in the system and the imagery they view.

Movie Night at End Point

This digital chamber can be considered a sort of VR movie theater, where an audience can enjoy the same content, and probably the same bucket of popcorn! While this setup makes the Liquid Galaxy a natural fit for any sort of exhibit, many End Point employees have also watched full length feature movies on the system during our monthly Movie Night at our Headquarters office in Manhattan. This sort of shared experience is not something that is possible on typical VR, because unlike VR the Liquid Galaxy is serving a larger audience and presenting stories in a more interactive way.

For most meetings, exhibitions, and other special occasions, the Liquid Galaxy helps to provide an amazing and impactful experience to the audience. Any scenario can be built for users to explore, and geospatial data sets can be presented immersively.

With the ability to serve a group of people simultaneously, Liquid Galaxy increases the impact of content presentation and brings a revolutionary visual experience to its audiences. If you’d like to learn more, you can call us at 212-929-6923, or contact us here.

Shrink XFS partition: Almost possible with LVM

2015-01-29T00:00:00+00:00

If you happen to have reached this page because you’re trying to shrink an XFS filesystem let’s put things straight: sorry, that’s not possible.

But before you go away you should know there’s still hope and I’d like to show you how I used a little workaround to avoid reinstalling a Red Hat Enterprise Linux 7 or CentOS 7 VM using XFS dump/restore on-the-fly and LVM capabilities, both standard choices for the regular RHEL/CentOS 7 setup.

First of all let’s clarify the situation I found myself in. For various reasons I had a CentOS 7 VM with everything already configured and working, installed not many days ago to test new software we’re evaluating.

The VM itself is hosted on a dedicated server we manage on our own, so I had a certain degree of freedom to what I could do without the need of paying any additional fee. You may not be in this same situation, but you can probably try some similar solution for little money if you’re using an “hourly-billed” VPS provider.

The problem was that, even if everything was working and configured, the virtual hard disk device attached to the machine was too big and on the wrong storage area of the virtualization hosting server.

There was also another minor glitch: The VM itself was using an old virtualization device driver (IDE-based) instead of the new VIRTIO one. Since I knew that the virtualized OS CentOS 7 is capable of using VIRTIO based devices I also took the chance to fix this.

Unfortunately, XFS is not capable of being shrunk at the moment (and for the foreseeable future) so what I needed to do was to:

add a new virtual device correctly using VIRTIO and stored in the right storage area of the virtualization host server
migrate all the filesystems to the new virtual device
set the VM OS to be working from the newly-built partition
dismiss the old virtual device

In my specific case this translated to connect to the virtualization hosting server, create a new LVM logical volume to host the virtual disk device for the VM and then add the new virtual device to the VM configuration. Unfortunately in order to have the VM see the new virtual device I had to shut it down.

While connected to the virtualization host server I also downloaded and attached to the VM the latest ISO of SysRescueCD which is a data rescue specialized Linux distribution. I’m specifically using this distro since it’s one of the few which offers the XFS dump/restore tools on the live ISO.

Now the VM was ready to be booted with the SysRescueCD Live OS and then I could start working my way through all the needed fixes. If you’re doing something similar, of course please make sure you have offsite backups and have double-checked that they’re readable before doing anything else.

First of all inspect your dmesg output and find what is the source virtual device and what’s the new target virtual device. In my case the source was /dev/sda and the target was /dev/vda

dmesg | less

Then create a partition on the new device (eg: /dev/vda1) as Linux type for the /boot partition; this should be of the same size as the source /boot partition (eg: /dev/sda1) and dedicate all the remaining space to a new LVM type partition (eg: /dev/vda2)

fdisk /dev/vda
# [create /boot and LVM partitions]

You could also mount and copy the /boot files or re-create them entirely if you need to change the /boot partition size. Since I kept /boot exactly the same so I could use ddrescue (a more verbose version of classic Unix dd).

ddrescue /dev/sda1 /dev/vda1

The next step is supposed to migrate the MBR and should be working but in my case the boot phase kept failing so I also needed to reinstall the bootloader via the CentOS 7 rescue system (not covered in this tutorial but briefly mentioned near the end of the article).

ddrescue -i0 -s512 /dev/sda /dev/vda

Then create the target LVM volumes.

pvcreate /dev/vda2
vgcreate fixed_centos_VG /dev/vda2
lvcreate -L 1G -n swap fixed_centos_VG
lvcreate -l 100%FREE -n root fixed_centos_VG
vgchange -a y fixed_centos_VG

Create the target XFS filesystem.

mkfs.xfs -L root /dev/fixed_centos_VG/root

And then create the swap partition.

mkfs.swap /dev/fixed_centos_VG/swap

Next create the needed mountpoints and mount the old source and the new empty filesystems.

mkdir /mnt/disk_{wrong,fixed}
mount /dev/fixed_centos_VG/root /mnt/disk_fixed
vgchange -a y wrong_centos_VG
mount /dev/centos_VG/root /mnt/disk_wrong

Now here’s the real XFS magic. We’ll use xfsdump and xfsrestore to copy the filesystem content (files, directory, special files) without having to care about files permission, type, extended ACLs or anything else. Plus since it’s only moving the content of the filesystem we won’t need to have a partition of the same size and it won’t take as long as copying the entire block device as the process will just have to go through real used space.

xfs_dump -J - /mnt/disk_wrong | xfs_restore -J - /mnt/disk_fixed

If you want a more verbose output, leave out the -J option. After the process is done, be sure to carefully verify that everything is in place in the new partition.

ls -lhtra /mnt/disk_fixed/

Then unmount the disks and deactivate the LVM VGs.

umount /mnt/disk_{old,fixed}
vgchange -a n centos_VG
vgchange -a n fixed_centos_VG

At this point in order to avoid changing anything inside the virtualized OS (fstab, grub and so on), let’s remove the old VG and rename the newer one with the same name the old one had.

vgremove centos_VG
pvremove /dev/sda2
vgrename {fixed_,}centos_VG

You should be now able to shutdown the VM again, detach the old disk and start the new VM which will be using the new smaller virtual device.

If the boot phase keeps failing, boot the CentOS installation media in rescue mode and after chroot-ing inside your installation run grub-install /dev/vda (targeting your new main device) to reinstall grub.

Only after everything is working as expected, proceed to remove the old unneeded device and remove it from the virtualization host server.

Easier IE Site Testing With RemoteIE

2014-12-02T00:00:00+00:00

Microsoft recently announced a new service which I’m finding very useful. RemoteIE lets you test your sites with IE (currently version 11) on Windows 10 Technical Preview. The service runs in Azure RemoteApp which is available for several clients including Android, iOS and Windows Phone. What’s great about this is that you do not have to install and maintain your own virtual machine with VirtualBox or VMWare.

To use RemoteIE you’ll need a valid Microsoft account—it’s easy to sign up if you don’t already one. Once you have an account and have downloaded & installed the Azure RemoteApp client of your choice it’s just a matter of starting it up and logging in. Happy Testing!

Test Web Sites with Internet Explorer for Free

2012-09-26T00:00:00+00:00

Browser Testing

While many Web Developers prefer to build sites and web applications with browsers like Chrome or Firefox it’s important for us to keep an eye on the browser market share for all web users. Internet Explorer (IE) still owns a large piece of this pie and because of this, it is important to test sites and applications to ensure they work properly when viewed in IE. This poses a potential problem for developers who do not use Windows.

Although I use OS X on my desktop, I have Windows virtual machines with IE 6,7,8,9 and 10 installed. I also have a Linux virtual machine running Ubuntu so I can check out Chrome/Chromium and Firefox on that platform. In the past I had tried solutions like MultipleIEs but wasn’t satisfied with them. In my experience I’ve found that the best way to see what users are seeing is to have a virtual machine running the same software they are.

I did some IE8 testing for a colleague a short time ago and suggested she should give VirtualBox a shot. Her response was “You should write a blog post about that!”. So here we are.

Free Tools

VirtualBox is a free virtualization application similar to Parallels or VMWare. These applications aim to solve this problem by allowing us to run different “guest” operating systems on the “host” operating system of our choice. For example, I run several versions of IE and Windows on my computer which is running OS X.

Microsoft have begun in recent years to release Virtual PC (VPC) images for free to enable Web Developers to test their sites and applications with various versions of Internet Explorer. The most recent images were released on July 30, 2012 and you can check them out here. Although these images are created to run on Microsoft’s Virtual PC application, with a little bit of effort we can also use them on Linux, OS X or Windows via VirtualBox.

For our purposes, we will be building a Windows 7 virtual machine running Internet Explorer 8. However, there are several images available with various Windows OS and Internet Explorer version combinations:

Windows 7 / IE8
Windows 7 / IE9
Windows Vista / IE7
Windows XP / IE6

There will likely be some differences creating virtual machines for the various images but the process will be similar to what we’ll document here for Windows 7 / IE.

Linux and OS X Users

In researching this article I came across a very helpful project on GitHub. The ievms project by xdissent is a bash script that automates this process. I was able to run the script and run my newly created Windows 7 / IE virtual machine shortly afterward.

Requirements

To start we’ll need to download and install VirtualBox. Be sure you have the most recent version available as it gets updated quite often. We’ll also need to install the VirtualBox Extension Pack. Both are available on the VirtualBox Downloads page.

Because the VPC images are quite large (~2.6GB in our case) they have been split up into several files. For the Windows 7 / IE8 image we’ll download the following files:

Once the files have been extracted we have created our virtual machine the disk will take up close to 10 GB. Make sure you have enough disk space available. I run my virtual machines on an external USB hard drive which works well if your local hard drive is starved for space.

Extract the VPC Files

Once you have all of the the VPC image files downloaded you’ll need to extract them:

Windows users: double-click the Windows_7_IE8.part01.exe file

Linux / OS X users: You will need to install unrar. I did so on OS X with Homebrew and there are packages available for Ubuntu as well.

Once you have unrar installed, issue the following command in the directory containing all of the VPC image files:

unrar e -y Windows_7_IE8.part01.exe

This will extract the files and combine them into a single .vhd file. The -y flag tells unrar to say “yes” to the EULA from Microsoft so be aware of that.

Create a new Machine

Once the .vhd file is ready we can create a new machine in VirtualBox. Start up VirtualBox and click the “New” button:

Click “Continue” to proceed.

VM Name and OS Type

We’ll enter a name (IE8 / Windows 7) and select “Microsoft” and “Windows 7” in the Operating System and Version drop-downs:

Click “Continue” to proceed.

Memory

Specify how much memory (RAM) you’d like to allocate to the virtual machine. This really depends on how much you have available but it is best to go with the minimum at the very least:

Click “Continue” to proceed.

Disk

Select the “Do not add a virtual hard drive” option.:

The reason we skip this for now is because we want to add the hard disk as an IDE drive. For some reason VirtualBox adds the disk as a SATA drive which causes the blue screen of death (BSOD) to appear when the virtual machine is booted up.

Click “Continue” to proceed and click “Continue” again when presented with the warning about not attaching a hard disk.

Review Settings and Create the VM

At this point you can review the settings you’ve made and click “Create” to complete the process:

Configure Storage Settings

Highlight your newly created virtual machine and click the “Settings” button. From there, click the “Storage” tab and add a hard disk to the IDE controller:

Absolute Pointing Device

In the Settings, click on the “System” tab and uncheck “Enable absolute pointing device”. I had to do this in order to to get the mouse working properly.

Start it Up

We’re almost there: next we need to start up the VM by clicking the “Start” button. You’ll see Administrator and IEUser accounts and the password for both is “Password1”. Log in as Administrator and check it out! It’s important to note when the virtual machine has the keyboard and mouse focus. When you click inside the virtual machine window, it will capture the input from your mouse and keyboard. To return focus back to your computer you will need to press the “host key” sequence on your computer:

The “host-key” sequence can be configured in the VirtualBox preferences. On my computer, the host-key sequence is the left CTRL key (displayed as Left ^). When the arrow icon is illuminated, the guest computer – Windows 7 in this case, has the focus. When I press the left CTRL key on my computer the arrow icon goes tray to indicate focus has been returned to my computer.

Activation

As soon as you log in you will be presented with the Windows Activation wizard. Follow the steps to activate this copy of Windows. Once that is complete you will be prompted to restart your computer. Go ahead and do that.

The screen resolution in the virtual machine will be quite low at first — I believe 800x600 is the default. Don’t worry about this as it will be addressed in the next step.

Guest Additions

Log in as Administrator (Password1) and Install the VirtualBox Guest Additions. To do this, navigate to Devices in the VirtualBox menu on your computer (the “host” machine"). Choose the menu option to “Install Guest Additions”. Each time you are prompted with the “Would you like to install…” dialog, choose “Install”:

This will install some drivers optimized by VirtualBox which allow you to change the screen resolution, improve the interaction between your host computer and the Windows guest operating system. Once the install process is complete you will be prompted to restart your virtual machine once more.

Finished Product

With Windows activated and the Guest Additions installed you should be able to log in to your very own Windows 7 / IE8 testing machine!

This free copy of Windows will operate for 30 days and the trial period can be extended twice by running the following at the command prompt:

slmgr –rearm

Check out the documentation at Microsoft for more details.

Automated VM cloning with PowerCLI

2012-07-23T00:00:00+00:00

Most small businesses cannot afford the high performance storage area networks (SANs) that make traditional redundancy options such high availability and fault tolerance possible. Despite this, the APIs available to administrators of virtualized infrastructure using direct attached storage (DAS) make it possible to recreate many of the benefits of high availability.

High Availability on SAN vs DAS

A single server failure in a virtualized environment can mean many applications and services can become unavailable simultaneously; for small organizations, this can be particularly damaging. High availability with SANs minimize the downtime of applications and services when a host fails by keeping virtual machine (VM) storage off the host and on the SAN. VMs on a failed host can then be automatically restarted on hosts with excess capacity. This of course requires SAN infrastructure to be highly redundant, adding to the already expensive and complex nature of SANs.

Alternatively, direct attached storage (DAS) is very cost effective, performant, and well understood. By using software to automate the snapshot and cloning of VMs via traditional gigabit Ethernet from host to host, we can create a “poor man’s” high availability system.

It’s important for administrators to understand that there is a very real window of data loss that can range from hours to days depending on the number of systems backed up and hardware in use. However, for many small businesses who may not have trustworthy backups, automated cloning is an excellent step forward.

Automated cloning with VMWare’s PowerCLI

Although End Point is primarily an open source shop, my introduction virtualization was with VMWare. For automation and scripting, PowerCLI, the PowerShell based command line interface for vSphere, is the platform on which we will build. The process is as follows:

A scheduled task executes the backup script.
Delete all old backups to free space.
Read CSV of VMs to be backed up and the target host and datastore.
For each VM, snapshot and clone to destination.
Collect data on cloning failures and email report.

I have created a public GitHub repository for the code and called it powercli_cloner.

Currently, it’s fairly customized around the needs of the particular client it was implemented for, so there is much room for generalization and improvement. One area of improvement is immediately obvious: only delete a backup after successfully replacing it. Also, the script must be run as a Windows user with administrator vSphere privileges, as the scripts assumes pass-through authentication is in place. This is probably best for keeping credentials out of plain text. The script should be run during non-peak hours, especially if you have I/O intensive workloads.

Hopefully this tool can provide opportunities to develop backup and disaster recovery procedures that are flexible, cost-effective, and simple. I’d welcome pull requests and other suggestions for improvement.

UTOSC 2012 talks of interest

2012-05-18T00:00:00+00:00

It’s been two weeks now since the Utah Open Source Conference for 2012. My fellow End Pointers wrote previously about it: Josh Ausborne about the mini Liquid Galaxy we set up there for everyone to play with, and Josh Tolley with a write-up of his talks on database constraints and KML for geographic mapping markup.

There were a lot of interesting talks planned, and I could only attend some of them. I really enjoyed these:

Rob Taylor on AngularJS
Brandon Johnson on Red Hat’s virtualization with oVirt, Spacewalk, Katello, and Aeolus
Clint Savage about RPM packaging with Mock & Koji
Daniel Evans on testing web applications with Capybara, embedded WebKit, and Selenium (which End Pointer Mike Farmer wrote about here back in December)
Aaron Toponce on breaking full-disk encryption (I missed this talk, but learned about it from Aaron in the hallway track and his slides afterwards)
Matt Harrison’s tutorial Hands-on intermediate Python, covering doctest, function parameters and introspection, closures, function and class decorators, and more.

I gave a talk on GNU Screen vs. tmux, which was fun (and ends with a live demo that predictably fell apart, and audience questions that you can’t hear on the recording). Here’s the video:

Follow this Direct YouTube link in case the embedded version doesn’t work for you. And here are the presentation slides.

I have a bit more to cover from the conference later!

Problem with Cisco VPN on Ubuntu 12.04

2012-05-07T00:00:00+00:00

A couple of days ago I had to change my notebook. I installed Ubuntu 12.04 on the new one, while on the previous one there was Ubuntu 11.10. There were no problems with copying all the files from the old to the new machine, including GPG and SSH keys. Everything went smoothly and I could connect to all the machines I needed.

The only problem was with VPN. While working for one of our clients, I need to connect to their VPN. On the old machine I did that through the Network Manager. Nothing easier, I went to the Network Manager, chose the Export option and saved all the settings to a file. I copied the file to the new computer and loaded it into the Network Manager.

The file loaded correctly. I could switch the VPN on. It said everything works. But in fact it didn’t. The message was “VPN is connected”, I could switch it on and off, but I couldn’t access any of the client’s resources available from my previous notebook.

The first thing I checked was the content of /etc/resolv.conf on both computers. The file without connecting to VPN looked like this on both computers:

$ cat /etc/resolv.conf
# Generated by NetworkManager
nameserver 127.0.0.1

When I connected to the VPN the files on both computers were quite different. For example on my new computer (and Ubuntu 12.04) the content of the file looked like this:

$ cat /etc/resolv.conf
# Generated by NetworkManager
domain something.net
search something.net
nameserver 127.0.0.1

I changed the data a little bit of course, so the domain names and IP addresses (except for 127.0.0.1) are not real.

On my old computer the resolv.conf file had a lot more entries, however I thought the above file should work as well. The problem was still the same: I couldn’t connect to the client’s resources.

The client is using the CISCO VPN, so I had to install network-manager-vpnc. This is just a plugin for network-manager which uses the vpnc program internally. I thought that maybe the plugin was doing something wrong.

I checked the plugin versions. Yes, they really differ. I started thinking about using the program without the Network Manager.

It turned out to be very simple to use. I need just a config file. The file is really simple:

IPSec gateway   something.net
IPSec ID        something.id
IPSec secret    somethingpass
Xauth username  mylogin
Xauth password  mypass

I keep all my local scripts in ~/bin (which can also be accessed as /home/szymon/bin). The directory ~/bin is added to the PATH environment variable. This way I can access all the scripts placed there in the console without providing the whole path. I did it by adding the following line at the end of my local ~/.bashrc file.

PATH=$PATH:$HOME/bin

To keep the things together I saved the config file at the same location ~/bin/vpn.conf.

Now I can connect to the VPN using:

$ sudo vpnc-connect /home/szymon/bin/vpn.conf

I can also stop the VPN using:

$ sudo vpnc-disconnect

To automate it a little bit I created a simple script stored at ~/bin/vpn:

#!/usr/bin/env bash

case "$1" in

start)
  sudo vpnc-connect /home/szymon/bin/vpn.conf
  ;;
stop)
  sudo vpnc-disconnect
  ;;
status)
  ps uaxf | grep vpnc-connect | grep -v grep
  ;;
restart)
  sudo vpnc-disconnect
  sudo vpnc-connect /home/szymon/bin/vpn.conf
  ;;
*)
  echo "Usage: vpn (start|stop|status|restart)"
  exit 1
  ;;

esac

This way I can simply write:

$ vpn start
[sudo] password for szymon:
VPNC started in background (pid: 13771)...

I noticed that now the /etc/resolv.conf file contains different entries than when using the Network Manager plugin:

$ cat /etc/resolv.conf
#@VPNC_GENERATED@ -- this file is generated by vpnc
# and will be overwritten by vpnc
# as long as the above mark is intact
# Generated by NetworkManager
nameserver 1.2.3.4
nameserver 1.2.3.4
search something.net

I can also disconnect from the VPN with simple command:

$ vpn stop
Terminating vpnc daemon (pid: 13771)

I’m using this script for a couple of days and I don’t have any problems with the CISCO VPN. It seems like the vpnc program in Ubuntu 12.04 is OK, however there is something wrong with the Network Manager plugin for vpnc.

Xen MAC mismatch VNC mouse escape HOWTO

2010-03-30T00:00:00+00:00

This is a story that probably shouldn’t need to be told if everything is in documentation somewhere. I’m not using any fancy virtualization management tools and didn’t have an easy time piecing everything together, so I thought it’d be worth writing down the steps of the manual approach I took.

Dramatis personæ:

Server: Red Hat Enterprise Linux 5.4 with Xen kernel
Guest virtual server: CentOS 5.4 running paravirtualized under Xen
Workstation: Ubuntu 9.10

The situation: I updated the CentOS 5 Xen virtual guest via yum and rebooted to load the new Linux kernel and other libraries such as glibc. According to Xen as reported by xm list, the guest had started back up fine, but the host wasn’t reachable over the network via ping, http, or ssh, including from the host network.

The guest wasn’t using much CPU (as shown by xm top), so I figured it wasn’t just a slow-running fsck during startup. And I was familiar with the iptables firewall rules on this guest, so I was fairly sure I wasn’t being blocked there. I needed to get to the console to see what was wrong.

The way I’ve done this before is using VNC to access the virtual console remotely. The Xen host was configured to accept VNC connections on localhost, which I could see by looking in /etc/xen/xend-config.sxp:

(vnc-listen '127.0.0.1')

There are 11 Xen guests, with consoles listening on TCP ports 5900-5910. Which one was the one? I don’t know any simple way to get a list that maps ports to Xen guests, but I did it this way:

ps auxww | grep qemu-dm

I noted the PID of the process that was running for my guest as revealed in its command line. Then I looked for the listener running under that PID:

netstat -nlp

I looked for $pid/qemu-dm in the PID/Program Name column and could then see the TCP port in the Local Address column. In my case it was 127.0.0.1:5903.

So I set up an ssh tunnel to the server for my VNC traffic:

ssh -f -N -L 5903:localhost:5903 root@$remote_host &

Then I opened the default Ubuntu/GNOME VNC viewer, labeled “Remote Desktop Viewer” under the Internet menu. This program is actually called Vinagre, and is basic but works. I connected to localhost:5903, since I’d forwarded my own local TCP port 5903 to the remote port 5903.

The remote console came up, and I was presented with the login banner and prompt. If I hadn’t had the root password, I would have needed to reboot the guest and start it in runlevel 0 to get a root shell without a password and change the password. But I did have the password, so that wasn’t necessary.

Then when trying to change to another window on my desktop, I ran into the biggest snag of the whole exercise: Getting control of the mouse out of the VNC remote desktop window and back in my own desktop! I couldn’t find anything accurate on this in any documentation, forums, etc. Finally I stumbled across the trick: Press F10, which pulls down the Machine menu in Vinagre and as a side-effect takes control of the mouse away from the remote desktop. It was nice not to have to ssh in from another machine to kill Vinagre. But it makes me wonder how I’d send an F10 through to the remote console …

Armed with the root password I was able to log into the guest and discover that only the lo (loopback) interface started on boot. The eth0 and eth1 interfaces failed because there was no virtual NIC available with the MAC addresses specified in /etc/sysconfig/network-scripts/ifcfg-eth0 and eth1.

That was because the virtual machine image had been cloned from another one and hadn’t been given new unique MAC addresses. The problem was easily fixed by updating the ifcfg-eth0 and ifcfg-eth1 files with the MAC addresses actually given to the interfaces, as seen by ifconfig, which were ultimately assigned by the Xen host in /etc/xen/$host in the vif parameter. (You can also specify no MAC addresses in the guest at all and it will use whatever it gets.)

Then after running service network restart the networking was back, and I rebooted to make sure it started correctly on its own.