Camping in the Clouds with Terraform and Ansible

2019-02-05T00:00:00+00:00

Photo by Andrew E. Larsen · CC BY-ND 2.0

Right, so, show of hands: How many of you work on some bit of web code by doing a git clone to your own laptop, developing the feature or bug fix, running through manual testing of the app until you’re happy with it, and off it goes back up to the repo when done? I’m curious, and I have a few questions for you:

Have you ever had a bit of code that worked locally, but didn’t in production because of some difference in systems, dependencies, or something else in the stack?
How do you show off your work to a client or management for approval? Can you demo several alternate changes to the same site at the same time?
How do you bring in coworkers to “look over your shoulder“ and help with something, especially ones that are far away?
How do you get a new coworker up to speed if they’re doing development themselves?
If you’re working on multiple things, do you create multiple clones?
How’re your backups?

Are you fidgeting nervously thinking about that? Sorry. ☹ But also, check out this little thing: DevCamps. It’s been an End Point staple for quite a while now, so if you’ve read our blog before you might have heard about it.

Long story short: In addition to any local development you do, this system will spin up your own little environments (“camps”) on a remote development server. Each camp includes a checkout of the code, separate httpd/nginx and app processes, and a dedicated database with a clone of the data. What’s that all mean?

Well, figuring that dev server is configured the same, you’ll be working right in a stack identical to production.
Each camp gets its own port, so you can link your beautiful results to someone and they’ll see exactly what the site will look like when deployed.
Bring coworkers in to a shared tmux session for code review or pair programming.
And if they just can’t resist working on the same project they can create their own camp, without having to install dependencies and get the stack operational locally.
Create multiple camps so you can work on new projects when in-flight projects get put on the back burner.
Those server backups your ops team promises are working will automatically preserve your work.
And unicorns become real.

Recently a client brought their app into “The Cloud.“ Kind of. No Kubernetes there, so I can’t use that coveted “k8s” blog post tag yet. It’s still a Java app sitting on a single big VM, but it’s still technically running in the cloud. So we used that opportunity to redo how this “campsite” works and made it cloudy, too. Each camp is a dedicated set of VMs that we spin up (and, for cost savings, shut down) as needed.

We do that using Terraform. We’re already using that for this client’s production systems. Terraform has a concept of workspaces, which basically clones your single set of systems and lets you apply that multiple times over. You can name the workspace pretty much anything, but we’ve stuck with the traditional camp method of just assigning numbers. Thus, each camp is a numbered workspace, with an identical VM for the app and an identical VM for the database. That gets handed over to Ansible for the same provisioning process.

The configuration split is actually a little more complex than that. We have a ‘prod’ and a ‘dev’ module, which is mostly just different instance sizes. Then, combined with appropriate DNS naming parameters, ‘camp’, ‘staging’, and ‘prod’ configs. Just pull from those as needed. I’ll probably take you into that in more depth in a later post.

But, instead of one big server which is a fixed cost no matter how many camps we’re running, each cloud camp adds to the cost only as long as it’s running. So, like I said above, shutting down the camps is a key point of this. We run them on smaller instances generally, but if you run a number of camps at once that cost adds up fast. Shutting down the camps when they’re not needed saves money. There’s a break-even line somewhere in there, and I’ll leave calculating that as an exercise for the reader. But there are other benefits, too:

On traditional camps running on a single system, they all share the same IP address and each gets its own port number. But running on multiple systems with multiple addresses you don’t have to worry about clients with odd egress firewalls that block traffic to nonstandard ports.
Filesystem paths shouldn’t matter inside apps, but sometimes they do, and running camps in their own VMs means the apps can be deployed exactly where they are in production.
Same with database ports: You get to talk to the system Postgres on its default port, and potentially even on a separate host.
We tend to run our monolithic camp servers with SELinux disabled, but with cloud camps you can run with SELinux enforcing in staging environments just like in production.

Cloud camps take you the rest of the way there. You can feel like you’re working and testing right on production without really feeling all that dirty about it. And, when you’re nearly done, you can control the resources assigned to the VMs, for example to switch the sizing configuration so that smaller VMs you use for development briefly become production-sized for load testing. Briefly.

I do love making things open source, but in this case I’m not sure there’s really anything to publish; it’s just the same Terraform configuration with just a little bit of tuning to make workspaces work better. But you’ve heard enough from me today, so I’ll write more about all those details later.

There, I hope I’ve planted some ideas in your mind about how you can change that development methodology for the better. On the whole we work better collaboratively, and camp-like systems for development can really assist that. It can feel weird and cumbersome at first, but the clients we have that have adopted it are generally happy campers.

It’s still early on, but our cloud-based camp experiment is showing lots of promise, too. Keep it in mind when you’re starting a development project.

2015 Perl Dancer Conference videos

2015-12-30T00:00:00+00:00

The 2015 Perl Dancer Conference has recently released the presentation videos. This year the conference was hosted in beautiful Vienna, Austria. Josh Lavin and I were both honored to attend the conference as well as give talks. Earlier, Josh wrote summaries of the conference:

Conference Recap

Conference Presentations

SpaceCamps “The Final Frontier”

I gave a talk exploring new technologies for End Point’s own DevCamps development tool. During the presentation I detailed my research into containers and what a cloud-based development environment might look like.

SpaceCamps Presentation Video

AngularJS & Dancer for Modern Web Development

Josh detailed his experience migrating legacy applications utilizing Dancer, AngularJS, and modern Perl techniques. Josh highlighted the challenges he faced during the process, as well as lessons he learned along the way.

AngularJS & Dancer for Modern Web Development Presentation Video

Lightning Talks

Josh and I both gave short “lightning talks.” Josh’s was on Writing Unit Tests for a Legacy App (Interchange 5), and mine was on Plack & Interchange 5.

To review the rest of the presentations please checkout the Perl Dancer Conference YouTube channel.

Summary

The Perl Dancer community continues to flourish and the conference this year hosted a record 5 core Dancer developers. Dancer is about to release the finalized version of its long awaited plugin infrastructure for Dancer2. A lot of work on this was completed during the conference. Being an organizer of the conference, it brings me great joy to see this success. This news along with the release of Perl 6, I am certain 2016 will be a wonderful year for not only Dancer but the entire Perl community.

Handling databases in dev environments for web development

2015-04-21T00:00:00+00:00

One of the biggest problems for web development environments is copying large amounts of data. Every time a new environment is needed, all that data needs to be copied. Source code should be tracked in version control software, and so copying it should be a simple matter of checking it out from the code repository. So that is usually not the problem. The main problem area is database data. This can be very large, take a long time to copy, and can impact the computers involved in the copy (usually the destination computer gets hammered with IO which makes load go high).

Often databases for development environments are created by copying a database dump from production and then importing that database dump. And since database dumps are text, they can be highly compressed, which can result in a relatively small file to copy over. But the import of the dump can still take lots of time and cause high load on the dev computer as it rebuilds tables and indexes. As long as your data is relatively small, this process may be perfectly acceptable.

Your database WILL get bigger

At some point though your database will get so big that this process will take too long and cause too much load to be acceptable.

To address the problem you can try to reduce the amount of data involved by only dumping a portion of the database data instead of all of it, or possibly using some “dummy sample data” instead. These techniques may work if you don’t care that development environments no longer have the same data as production. However, one serious problem with this is that a bug or behavior found in production can’t be replicated in a development environment because the data involved isn’t the same. For example, say a customer can’t checkout on the live site but you can’t replicate the bug in your development environment to fix the bug. In this example, the root cause of the problem could be a bug in the code handling certain products that are out of stock, and since the dev database didn’t have the same data it could make finding and fixing these types of problems a lot harder.

Snapshots

Another option is to use file system snapshots, like LVM snapshots, to quickly make clones of the database without needing to import the database dump each time. This works great if development environments live on the same server, or at least the development databases live on the same server. You would need to create a volume to hold a single copy of the database; this copy would be the origin for all snapshots. Then for each development environment, you could snapshot the origin volume, mount it read-write in a place accessible by the developer, customize the database configuration (like setting a unique port number to listen on), and then start up the database. This then provides a clone of the entire database in a tiny fraction of the time and uses less disk space and other system resources too.

In using snapshots there are some things you’ll need to be careful about. Snapshots are usually created using copy-on-write tables. The more snapshots mounted read-write, the more IO overhead is involved for the volumes involved. For this reason it is important that writes to the origin volume be avoided as much as possible while the snapshots are open. Also, snapshots that get a lot of writes can fill up their copy-on-write table, and depending on the file system and database that you are using this can be a big problem. So it is important to monitor each open snapshot for how full it is and increase their size if needed so they don’t fill up. Updating the origin database will require shutting down and removing all snapshots first, then update the origin database, then create and mount all the snapshots again. This is because all the copy-on-write tables would get full if you tried to update the origin while the snapshots are open.

Using snapshots like this may sound more complicated, and it is, but the processes involved can be scripted and automated and the benefits can be pretty significant if you have several developers and a lot of data to copy.

Looking at development environments with DevCamps and Vagrant

2014-08-25T00:00:00+00:00

For most web developers, you have practices and tools that you are used to using to do your work. And for most web developers this means setting up your workstation with all the things you need to do your editing, compiling, testing, and pushing code to some place for sharing or deployment. This is a very common practice even though it is fraught with problems- like getting a database setup properly, configuring a web server, any other services (memcached, redis, mongodb, etc), and many more issues.

Hopefully at some point you realize the pain that is involved in doing everything on your workstation directly and start looking for a better way to do web development. In this post I will be looking at some ways to do this better: using a virtual machine (VM), Vagrant, and DevCamps.

Using a VM for development

One way to improve things is to use a local virtual machine for your development (for example, using VirtualBox, or VMware Fusion). You can edit your code normally on your workstation, but then execute and test it in the VM. This also makes your workstation “clean”, moving all those dependencies (like a database, web server, etc.) off your workstation and into the VM. It also gets your dev environment closer to production, if not identical. Sounds nice, but let’s break down the pros and cons.

Benefits of using a VM

Dev environment closely matches production.
Execute and test code in a dedicated machine (not your workstation directly).
Allows for multiple projects to be worked on concurrently (one VM per project).
Exposes the developer to the Operations (systems administration) side of the web application (always a good thing).
Developer can edit files using their favorite text editor locally on the workstation (but will need to copy files to the VM as needed).

Problems with using a VM

Need to create and configure the VM. This could be very time consuming and error prone.
Still need to install and configure all services and packages. This could also be time consuming and error prone.
Backups of your work/configuration/everything are your own responsibility (extremely unlikely to happen).
Access to your dev environment is extremely limited, thus probably only you can access it and test things on it. No way for a QA engineer or business owner to test/demo your work.
Inexperienced developers can break things, or change them to no longer match production (install arbitrary packages, different versions than what is in production, screw up the db, screw up Apache configuration, etc.).
If working with an established database, then downloading a dump, installing, and getting the database usable is time consuming and error prone. (“I just broke my dev database!” can be a complete blocker for development.)
The developer needs to set up networking for the VM in order to ssh to it, copy files back and forth, and point a web browser to it. This may include manually setting up DNS, or /etc/hosts entries, or port forwarding, or more complex setups.
If using SSL with the web application, then the developer also needs to generate and install the SSL cert and configure the web server correctly.

Vagrant

What is Vagrant? It is a set of tools to make it easier to use a virtual machine for your web development. It attempts to lessen many of the problems listed above through the use of automation. By design it also makes some assumptions about how you are using the VM. For example, it assumes that you have the source code for you project in a directory somewhere directly on your workstation and would prefer to use your favorite text editor on those files. Instead of expecting you to continually push updated files to your VM, it sets up a corresponding directory on the VM and keeps the two in sync for you (using either shared folders, NFS, Samba, or rsync). It also sets up the networking for accessing the VM, usually with port forwarding, so you don’t have to worry about that.

Benefits of Vagrant

Same as those listed above for using a VM, plus…
Flexible configuration (Vagrantfile) for creating and configuring the VM.
Automated networking for the VM with port forwarding. Abstracted ssh access (don’t need to set up a hostname for the VM, simply type vagrant ssh to connect). Port forwarded browser access to the VM (usually http://localhost:8080, but configurable).
Synced directory between your workstation and the VM for source code. Allows for developers to use their favorite text editor locally on their workstation without needing to manually copy files to the VM.
Expects the use of a configuration management system (like puppet, chef, salt, or bash scripts) to “provision” the VM (which could help with proper and consistent setup).
Through the use of Vagrant Cloud you can get a generated url for others to access your VM (makes it publicly available through a tunnel created with the command vagrant share).
Configuration (Vagrantfile and puppet/chef/salt/etc.) files can be maintained/reviewed by Operations engineers for consistency with production.

Problems with Vagrant

Still need to install and configure all services and packages. This is lessened with the use of a configuration management tool like puppet, but you still need to create/debug/maintain the puppet configuration and setup.
Backups of your work/configuration/everything are your own responsibility (extremely unlikely to happen). This may be lessened for VM configuration files, assuming they are included in your project’s VCS repo along with your source code.
Inexperienced developers can still break things, or change them to no longer match production (install arbitrary packages, different versions than what is in production, screw up the db, screw up Apache configuration, etc.).
If working with an established database, then downloading a dump, installing, and getting the database usable is time consuming and error prone. (“I just broke my dev database!” can be a complete blocker for development.)
If using SSL with the web application, then the developer also needs to generate and install the SSL cert and configure the web server correctly. This might be lessened if puppet (or whatever) is configured to manage this for you (but then you need to configure puppet to do that).

DevCamps

The DevCamps system takes a different approach. Instead of using VMs for development, it utilizes a shared server for all development. Each developer has their own account on the camps server and can create/update/delete “camps” (which are self-contained environments with all the parts needed). There is an initial setup for using camps which needs thorough understanding of the web application and all of its dependencies (OS, packages, services, etc.). For each camp, the system will create a directory for the user with everything related to that camp in it, including the web application source code, their own web server configuration, their own database with its own configuration, and any other resources. Each camp is assigned a camp number, and all services for that camp run on different ports (based on the camp number). For example, camp 12 may have Apache running on ports 9012 (HTTP) and 9112 (HTTPS) and MySQL running on port 8912. The developer doesn’t need to know these ports, as tools allow for easier access to the needed services (commands like mkcamp, re for restarting services, mysql_camp for access to the database, etc.).

DevCamps has been designed to address some of the pain usually associated with development environments. Developers usually do not need to install anything, since all dependencies should already be installed on the camps server (which should be maintained by an Operations engineer who can keep the packages, versions, etc. consistent with production). Having all development on a server allows Operations engineers to backup all dev work fairly easily. Databases do not need to be downloaded, manually setup, or anything- they should be set up initially with the camps system and then running mkcamp clones the database and sets it up for you. Running refresh-camp --db allows a developer to delete their camp’s database and get a fresh clone, ready to use.

Benefits of DevCamps

Each developer can create/delete camps as needed, allowing for multiple camps at once and multiple projects at once.
Operations engineers can manage/maintain all dependencies for development, ensuring everything is consistent with production.
Backups of all dev work is easy (Operations engineer just needs to backup the camps server).
Developer does not need to configure services (camp templating system auto-generates needed configuration for proper port numbers), such as Apache, nginx, unicorn, MySQL, Postgres, etc.
SSL certificates can be easily shared/generated/installed/etc. automatically with the mkcamp script. Dev environments can easily have HTTPS without the developer doing anything.
Developers should not have permissions to install/change system packages or services. Thus inexperienced developers should not be able to break the server, other developer’s environments, install arbitrary software. Screwing up their database or web server config can be fixed by either creating a new camp, refreshing their existing one, or an Operations engineer can easily fix it for them (since it is on a central server they would already have access to, and not need to worry about how to access some VM who knows where).

Problems with DevCamps

Since all camps live on a shared server running on different ports, this will not closely match production in that way. However, this may not be significant if nearly everything else does closely match production.
Adding a new dependency (for example, adding mongodb, or upgrading the version of Apache) may require quite a bit of effort and will affect all camps on the server- Operations engineer will need to install the needed packages and add/change the needed configuration to the camps system and templates.
Using your favorite text editor locally on your workstation doesn’t really work since all code lives on the server. It is possible to SFTP files back and forth, but this can be tedious and error prone.
Many aspects of the Operations (systems administration) side of the web application are hidden from the developer (this might also be considered a benefit).
All development is on a single server, which may be a single point of failure (if the camps server is down, then all development is blocked for all developers).
One camp can use up more CPU/RAM/disk/etc. then others and affect the server’s load, affecting the performance of all other camps.

Concluding Thoughts

It seems that Vagrant and DevCamps certainly have some good things going for them. I think it might be worth some thought and effort to try to meld the two together somehow, to take the benefits of both and reduce the problems as much as possible. Such a system might look like this:

Utilize vagrant commands and configuration, but have all VMs live on a central VM server. Thus allowing for central backups and access.
Source code and configuration lives on the server/VM but a synced directory is set up (sshfs mount point?) to allow for local editing of text files on the workstation.
VMs created should have restricted access, preventing developers from installing arbitrary packages, versions, screwing up the db, etc.
Configuration for services (database, web server, etc.) should be generated/managed by Operations engineers for consistency (utilizing puppet/chef/salt/etc.).
Databases should be cloned from a local copy on the VM server, thus avoiding the need to download anything and reducing setup effort.
SSL certs should be copied/generated locally on the VM server and installed as appropriate.
Sharing access to a VM should not depend on Vagrant Cloud, but instead should use some sort of internal service on the VM server to automate VM hostname/DNS for browser and ssh access to the VM.

I’m sure there are more pros and cons that I’ve missed. Add your thoughts to the comments below. Thanks.

Proxy Nginx ports using a regular expression

2014-03-19T00:00:00+00:00

I’m working on a big Rails project for Phenoms Fantasy Sports that uses the ActiveMerchant gem to handle Dwolla payments. One of the developers, Patrick, ran into an issue where his code wasn’t receiving the expected postback from the Dwolla gateway. His code looked right, the Dwolla account UI showed the sandbox transactions, but we never saw any evidence of the postback hitting our development server.

Patrick’s theory was that Dwolla was stripping the port number off the postback URL he was sending with the request. We tested that theory by using the RequestBin.com service for the postback URL, and it showed Dwolla making the postback successfully. Next, we needed to verify that Dwolla could hit our development server on port 80.

I started Nginx on port 80 of our dev server and Patrick fired his Dwolla transaction test again. The expected POST requests hit the Nginx logfile. Suspicions confirmed. It looked like we would just have to work around the Dwolla weirdness by proxying port 80 to the port that Patrick’s development instance was running on. Then we’d need a way to make that work for the other developers’ instances on the dev. server, as well.

Proxying a single port to another port with Nginx is easy, but that second requirement is a little more complicated. We did get a little lucky with this bit, however. We are using DevCamps.org “camps” for this project. DevCamps’ naming convention for a given development instance (AKA a “camp”) number uses a two digit camp number as the hostname and as part of the port number. For example, camp 42 would run on 42.camp.example.com:9042. (I bet you can already see where I’m going with this.)

I tweaked the Nginx config for the port 80 instance to use a regex to capture the hostname (“42” in this case) from the server_name portion of the HTTP request. It then appends that to the full hostname/IP address and the first two digits of that camp’s port number. That made the proxy work for everyone’s camp. Finally, I updated the config to work with any URI under the /dwolla directory.

We now have the following tidbit in our Nginx config:

server {
    # [SNIP unrelated config stuff]

    server_name ~^(?<portname>\d\d)\.camp\.;

    location /dwolla {
        proxy_pass        http://169.29.89.157:90${portname}$uri;
        proxy_set_header  X-Real-IP  $remote_addr;
    }
}

As root, I ran service nginx reload to pick up the new config changes. Now Nginx automagically proxies connections to specific ports based on the server’s hostname.

The more the merrier? Not always...

2014-02-05T00:00:00+00:00

Recently we were working on an image manipulation function for a customer that used ImageMagick’s convert command. We had spent tons of time working on the program and the associated system that was going to use it. Everything was working fine. The whole system was nice and snappy. Great job everyone!

Until we moved it into production where the page load times were 8 to 10 times slower than in our camps development system …

We all sprang into action to try to figure out why the page load times were so slow now. We reviewed system settings, configuration files for the database and application, and anything else we could think of. We made sure the OS packages, ImageMagick version, and various other things were the same, and everything looked right. The production hardware has twice the RAM and 4 times the number of processors as development does. So what the heck is going on?

To distract ourselves and hope for more insight, we tried to optimize the code a bit and while making it a bit better we were still 6 to 8 times slower than in development. We deactivated the section of the site overnight so we could sleep on it. Luckily this was a new product line so it wasn’t tragic to turn it off.

The next morning while discussing, a co-worker mentioned that the larger number of processors in production could be relevant. At first I was a bit taken aback because this goes against everything we think about more being better. How could it be that having 4 times the number of processors was a bad thing?!

Well, it turns out that ImageMagick is a threaded program and will use as many processors in parallel as it can to accomplish whatever task it is asked to do. In doing this instead of splitting up the convert over 8 processors, it was now splitting it up over 32 processors. The extra work it was taking to manage the work being split between those extra 24 processors actually greatly slowed down the processing work!

Luckily ImageMagick respects an environment variable setting to limit the maximum number of threads it will use. We set the OMP_NUM_THREADS environment variable to 4, re-ran the code, and then it performed as expected in production. It could be set to more or less threads, but we found that was fine and aside from keeping performance high, it keeps this one process from dominating CPU usage while it is running.

It was a very interesting riddle to solve so I figured I’d share it.

Ecommerce Innovation 2013

2013-10-16T00:00:00+00:00

Mark Johnson and I went to the Ecommerce Innovation 2013 conference in beautiful Hancock, NY. The event was hosted by Sam Batschelet of West Branch Resort. The conference was spread out over three days and was very well planned. We had plenty of time in between talks to mingle with the other people. All of the talks were very insightful and informative. I found the mixture of technology and marketing talks beneficial. I have already discussed some things with my clients that I learned.

A brief overview of the talks

Jure Kodzoman of Informa had two different subjects.
- His first talk was about Template::Flute which is a Perl based template system which is the default template for Interchange 6. It utilizes the use of html classes to figure out where to parse in the data returned from your Perl code. Overall it seems pretty straight forward to use.
- His second talk was about the current state of the new Interchange 6 demo store.
Ana Kozole of Informa had a talk named “Remarketing with Banners” that was really informative.The base of this is to have the ability to show specific banners to visitors on different websites. She discussed different remarketing techniques including creating specific lists based on different criteria like all visitors, people who got to the checkout page but didn’t checkout or people who used a coupon code etc. You can also use remarketing lists for search ads.
Luka Klemenc of Informa gave two talks.
- He discussed some a CRM system that they had developed in house and some of the pros and cons.
- Luka gave us short talk about the ways to know whether or not your newsletter is effective.
Josh Lavin of Perusion talked about a new template for Interchange 5 called Strap which is based on Bootstrap version 2.3.2. With this new template they have created a bunch of page and url changes to make the stock Interchange much more SEO friendly. This also makes a few underlying changes like assuming the username to login would be your email address and creating a multiple page checkout.
Mike Heins of Perusion gave us a brief history of Interchange and compared it to modern day frameworks like Dancer. He also gave us a brief overview of PCI compliance and how Interchange holds up. He introduced us to the features of the Perusion Payment Server which is a remote credit card processing system that helps with PCI compliance.
Stefan Hornburg of LinuXia Systems discussed two different topics with us.
- gave us an overview of where the Interchange 6 project currently is and where it’s going. He gave us some code samples of the way we can do simple things like add an item to the cart, fetch the subtotal of the cart and talk to the database.
- Stefan walked us through an integration he did for OpenERP with Interchange 5 using RPC::XML.
Mark Johnson gave us a review of the modifications to Interchange 5 to allow web servers like nginx or Apache to cache entire pages. He discussed how we modified Interchange 5 for a customer to help with a DDoS attack. He laid out all of the new usertags and directives you will need to set to get pages to be cachable including some “gotchas” like not sending cookie information if you want this page to be cached. We hope that this feature will be included in the Interchange 5.8.1 release.
Sam Batschelet gave a talk about DevCamps and the reasons why it is so great. He discussed things like using Perlbrew and Carton in camps to help get around the fact that most Linux operating systems ship with a pretty old version of Perl. He also expanded on a few features that we hope to get released soon.

The last big item on the schedule was a 2 hour round table discussion about the database schema for Interchange 6. It was a very good discussion with many different opinions for adjustments. Most of us based our suggestions on past experience with clients. I do not think we are finished making adjustments to it but we are on the right path to a very flexible setup.

Overall I thought the conference was a great success. It was great to meet in person some people I had only seen on a mailing list before and pass around ideas for the future. I cannot wait to see what cool new things we will have to discuss next year!

eCommerce Innovation Conference 2013

2013-09-20T00:00:00+00:00

The eCommerce Innovation Conference 2013 is a new conference being held in Hancock, New York, between October 8th and 11th. The conference aims to discuss everything ecommerce with a focus on Perl-based solutions including Dancer and Interchange. It isn’t geared directly to any one specific type of person unlike most conferences. The current speakers list include in-house ecommerce software developers, consultants, sales managers, project managers, and marketing experts. The talk topics range from customer relationship management to template engines for Perl.

Mark Johnson and I are both going to be speaking at the conference. Also there will be Mike Heins, creator of Interchange, and Stefan Hornburg, longtime Interchange development group “team captain”.

Mark is going to be discussing full page caching in Interchange 5. This is becoming a more frequent request from our larger customers. They want to be able to do full page caching to allow the web browser and a caching proxy server alone to handle most requests leaving Interchange and the database open to handle more shopping-based requests like add to cart or checkout. This is a commonly-used architecture in many application servers, and my colleague David Christensen has several new features already in use by customers to make full-page caching easier, which are expected to go into Interchange 5.8.1 soon.

I will be doing a talk on multi-site setup in Interchange 5. This is a request we have received frequently over the years. Companies may either already have some kind of wholesale website or just want to have multiple websites use the same database and programming but allow for different website designs. They normally need to control what website a product will show up on and possibly adjust the price accordingly. I’ll discuss the different methods we have used to accomplish this at End Point.

I see on the schedule that Sam Batschelet will be speaking about the camps system and some new capabilities he’s added for perlbrew and Carton, among other things. We are also using camps some places with perlbrew and plenv, so it will be interesting to compare notes. I hope we’ll see some discussion and/or contribution to the open source DevCamps project soon!

It promises to be a very nice conference with lots of diverse information!

Git as rsync

2013-02-14T00:00:00+00:00

I had a quick-and-dirty problem to solve recently:

The clients had uploaded many assorted images to a development camp, but the .gitignore meant those updates were not picked up when we committed and pushed and rolled out to the live site. Normally, one would just rsync the files, but for various reasons this was not practical.

So my solution, which I think can get filed under “Stupid ‘git’ tricks (as opposed to Tricks of a Stupid Git)”:

(on the source repo)

$ git checkout -b images_update
$ git add -f path-to-missing-images
$ git commit -m 'Do not push me! I'm just a silly temporary commit'

(“add -f” forces the images into the index, overriding our gitignore settings)

(on the target repo)

$ git remote add images /path/to/source/repo
$ git fetch
$ git checkout -f images/images_update path-to-missing-images
$ git remote rm images
$ git reset HEAD path-to-missing-images

That last “git reset” is because the newly-restored images will be git-added by default, and we didn’t want them committed to the central repo.

So what did we do here? For those dumbfounded by the level of silly, we used git to record the state of all the files in a certain path; then we pulled them back out into another location without disturbing anything already there. But as a benefit, we do have a record of what happened, in case we need to reproduce it.

Camp tools

2013-01-14T00:00:00+00:00

Devcamps are such a big part of my everyday work that I can’t imagine life without them. Over the years, I developed some short-cuts in navigating camps that I also can’t live without: I share them below.

function camp_top() {
  if [ -n "$1" ]
  then
      cd ~/camp${1}
  elif [[ $(pwd) =~ 'camp' ]]
  then
      until [[ $(basename $(pwd)) =~ '^camp[[:digit:]]+' ]]
      do
          if [[ $(pwd) =~ 'camp' ]]
          then
              cd ..
          else
              break
          fi
      done
  fi
}
alias ct='camp_top; pwd'

function cat_root() {
  camp_top $*
  cd catalogs/* >/dev/null
}
alias cr='cat_root; pwd'

function pages_root() {
  cat_root $*
  cd pages >/dev/null
}
alias pr='pages_root; pwd'

function what_camp() {
  c=$( camp_top $* 2> /dev/null; basename $( pwd ))
  echo $c
}

(“cat_root” and “pages_root” are very Interchange-specific; you may find other short-cuts more useful in your particular camp.)

There’s nothing terribly ground-breaking here, but if bash is not your native shell-tongue, then you might find these useful.

What I do is to stash these somewhere like “$HOME/.bash_camps”, then alter my .bashrc:

# Source campy definitions
if [ -f ~/.bash_camps ]; then
 . ~/.bash_camps
fi

That’s all it takes. Have you a camp-y shell script, function, or alias? Please share in the comments!

DevCamps: Creating new camps from a non-default Git branch

2012-08-31T00:00:00+00:00

I recently set up part of a new Rails project DevCamps installation with a unique Git repo setup and discovered a trick for creating camps from a Git branch other than master. Admittedly, the circumstances that led to me discovering this trick are a bit specific to this project, but the trick itself can be useful in other situations as well.

The Git repo specified in local-config had a master branch with nothing in it but the standard “initial commit.” This relatively new project uses a simplifed git-flow workflow and as such, all its code was still in the “develop” branch.

In my case, this empty-ish master branch meant there were no tracked files in CAMP_PATH/public directory. This meant that Git did not create that directory when the repo is cloned by mkcamp. This meant that apache2 would refuse to start. Camping without a web server makes my back hurt, so I snooped around a little bit…

I discovered two things:

You can tell git clone which branch to checkout initially by passing it a ‘–branch $your_non_default_branch’ switch
The mkcamp command will happily pass that switch (as well as any other spicy options you include) along to the git clone system command it executes. To do that, just add it to your camp type’s local-config file as part of the ‘repo_path_git’ config variable. For example:

repo_path_git:git@github.com:somegituser/somegitrepo.git –branch develop

Note that this option means your fresh new camp won’t have a ‘master’ branch checked out. This might confuse some users, but we all know the ‘master’ branch is nothing but a tracking branch with some convention mixed in. A simple git checkout master will create that expected master branch easily enough. It’s probably worth giving your devs a heads up about this, lest they think something wonky is afoot with mkcamp.

Now, there are people out there that may try to find fault with my solution. These detractors, these misanthropes, these malingering sluggards might cry “Why don’t you just commit an empty CAMP_PATH/public/.gitkeep” to your master branch?" Well, I like a clean git history. So, to those people I would say, “David, that’s messy and silly and wouldn’t make a very good blog article at all. I’m embarrassed for you for even bringing it up, David.”

The Berkman Center projects

2012-06-13T00:00:00+00:00

Brian Gadoury and his awesome socks gave a great and informative presentation on our client Berkman Center.

The Berkman Center’s mission is to explore and understand cyberspace; and to study its development, dynamics, norms and standards.

Brian’s first project was called Zone One that is a document repository tool that houses all of the electronic documents for use by Harvard. You can search by document name, tags, and content. Zone One is written using open source technologies such as Ruby on Rails, Postgres, Solr, and DevCamps.

The application includes an array of features that allow you to interact with the search results and the documents themselves. Each entry can have detailed meta-data associated with it, that can be referenced by future users.

One of the nice features of Zone One is its upload feature. It allows you to drag and drop files into a section the form for upload. The upload interface also includes a very nice SFTP system that utilizes virtual users on the system you are uploading to, to manage the files being uploaded and connected to the entry you are creating.

Phase two of the project will include features to export to external repositories, a bulk import feature, and an API.

DevCamps setup with Ruby 1.9.3, rbenv, Nginx and Unicorn

2012-02-10T00:00:00+00:00

I was working with Steph Skardal on the setup of a new DevCamps installation that was going to need to use Ruby 1.9.3, Rails 3, Unicorn and Nginx. This setup was going to be much different than a standard setup due to the different application stack that was required.

The first trick for this was going to get Ruby 1.9.3 on the server. We were using Debian Squeeze but that still only comes with Ruby 1.9.1. We wanted Ruby 1.9.3 for the increased overall speed and significant speed increase with Rails 3. We decided on using rbenv for this task. It’s a very easy to setup utility that allows you to maintain multiple version of Ruby in your system user account without the headache of adjusting anything but the PATH environment variable. It takes advantage of another easy to setup utility called ruby build to handle the actual installation of the Ruby source code.

A quick and easy version for setting up a user with this is as follows:

Ensure you are in the home directory. Then, clone the repository into a .rbenv directory

git clone git://github.com/sstephenson/rbenv.git .rbenv

Adjust your users path to find the newly installed commands

echo 'export PATH=$HOME/.rbenv/shims:$HOME/.rbenv/bin:$PATH' >> ~/.bash_profile

Install Ruby version 1.9.3-p0

rbenv install 1.9.3-p0

Make Ruby version 1.9.3-p0 your default version every time you log in

rbenv global 1.9.3-p0

Install the bundler gem for Ruby version 1.9.3-p0

gem install bundler

Refresh rbenv to let it know the new system command bundler exists

rbenv rehash

Now you are ready to use the bundler gem to install any other gems required for the application.

The normal camps setup assumes you are going to be using Apache for the web server. In this case, we wanted to use Nginx due to memory constraints. We decided to use the proxy capability and just proxy through to Unicorn instead of having to build our own version Nginx to use Passenger. To do this, we had to use a feature in the local-config file in camps that allows you to skip the Apache setup and use your own commands to start, stop and restart your web server and application. Here is the example from our local-config that controlls Nginx and Unicorn. This approach could also be used with Interchange or any other application if you need other services started when mkcamp is run.

skip_apache:1
httpd_start:/usr/sbin/nginx -c __CAMP_PATH__/nginx/nginx.conf
httpd_stop:pid=`cat __CAMP_PATH__/var/run/nginx.pid 2>/dev/null` && kill $pid
httpd_restart:pid=`cat __CAMP_PATH__/var/run/nginx.pid 2>/dev/null` && kill -HUP $pid || /usr/sbin/nginx -c __CAMP_PATH__/nginx/nginx.conf
app_start:__CAMP_PATH__/bin/start-app
app_stop:pid=`cat __CAMP_PATH__/var/run/unicorn.pid 2>/dev/null` && kill $pid
app_restart:pid=`cat __CAMP_PATH__/var/run/unicorn.pid 2>/dev/null` && kill $pid ; sleep 5 ;  __CAMP_PATH__/bin/start-app

The contents of the start-app script is simply.

cd __CAMP_PATH__ && bundle exec unicorn_rails -c __CAMP_PATH__/config/unicorn.conf.rb -D

You could create one script that handles all aspects of start, stop and restart if you wanted. This setup really wasn’t much harder than a normal Ruby on Rails setup. The added time here required to set up rbenv per camp user is offset by the fact that users can manage and try multiple versions of ruby.

Running Integration Tests in WebKit Without a Browser

2011-12-08T00:00:00+00:00

As your ruby web applications increase in UI complexity, they get harder to test using the standard Cucumber or even RSpec integration test suites. This is because of introduction of JavaScript in your UI. You’ve probably been there before. Here’s the use case: You’ve written your integration tests for your Rails app and up to this point, you’ve been able to get away with not tagging your cucumber scenarios with the “@javascript” tag. Everything is going smoothly and then it’s time to implement that one UI feature that is going to require an Ajax call or some javascript to hide or unhide a crucial piece of the user experience. This must be included in your integration tests.

So you go through the pain of setting up cucumber to work with selenium and tag your scenario as javascript so that it will run the test in the browser. At first, there’s this thrill of excitement as you get to see Firefox load, and then run through a series of steps, executing your tests and then seeing them pass. Job done.

But maybe there’s a different scenario at play here. What if you don’t do your development in an environment that has a browser? At End Point, we are strong advocates of doing development on the same environment that your app is going to run on. It eliminates unexpected issues down the road. We believe in it so much, actually, that we’ve created DevCamps that allows you to setup development environments on a server.

Obviously, your selenium based tests are not going to work here without some work to get it to run headless.

The good folks at thoughtbot have come up with a great solution to this and it is called capybara-webkit. It assumes that you are using capybara for your testing framework. If you are using webrat, the transition is fairly smooth. You’ll probably only need to change a few minor details in your tests.

What capybara-webkit does for you is enable you to run your tests inside of WebKit. This will simulate an environment that will be very close to what you would see in Google Chrome or Safari as well as many mobile browsers. I’ve found that except for some edge cases, it covers Firefox and IE as well.

To install capybara-webkit you will need to install the Qt development toolkit. It’s fairly straight forward so I’ll just refer you to the github wiki page for instructions for the various platforms. In Ubuntu, I just ran the following:

sudo apt-get install libqt4-dev

If you are installing on a server environment, you’ll also need to install Xvfb. You can do that in Ubuntu with the following command:

sudo apt-get install xvfb

It’s a little outside the scope of this blog post to go into other little things you need to setup with xvfb. The important thing is that you set it up to run on display 99. Another important note, is that you don’t have to set it up to run on boot. We will be starting it up when we run our tests if it isn’t running.

The next step is to configure your cucumber tests to use the capybara-webkit driver. To do that, add

gem "capybara-webkit"

to your Gemfile in the development and test group. Then in your env.rb file for cucumber add the following lines:

Capybara.javascript_driver = :webkit

In some cases, I’ve found it helpful to also specify a server port and app_host as follows:

Capybara.server_port = '8000'
Capybara.app_host = 'http://localhost:8000'

Now your tests are setup to run in WebKit. The final step is running the tests. To do this, you’ll need to run them from within xvfb. You can do that with the following command:

xvfb-run bundle exec cucumber

I’ve created an alias for this and dropped it in my .bashrc file. Here’s my entry, but you can set it up anyway you’d like.

alias xcuke="xvfb-run bundle exec cucumber"

Now running tests is a simple as running xcuke from the root of my Rails application.

There are a couple of big benefits to running capybara-webkit. First is speed. In my experience tests run much faster than they do in Selenium. Second, all JavaScript errors are dumped to STDOUT so you can see them in the output of your cucumber tests. Third, all of your tests are being run on WebKit instead of rack so you get a test environment that acts more like a real browser would behave.

Thanks to the guys at thoughtbot for putting together this awesome gem.

Semaphore limits and many Apache instances on Linux

2011-12-07T00:00:00+00:00

On some of our development servers, we run many instances of the Apache httpd web server on the same system. By “many”, I mean 30 or more separate Apache instances, each with its own configuration file and child processes. This is not unusual on DevCamps setups with many developers working on many projects on the same server at the same time, each project having a complete software stack nearly identical to production.

On Red Hat Enterprise Linux 5, with somewhere in the range of 30 to 40 Apache instances on a server, you can run into failures at startup time with this error or another similar one in the error log:

[error] (28)No space left on device: Cannot create SSLMutex

The exact error will depend on what Apache modules you are running. The “space left on device” error does not mean you’ve run out of disk space or free inodes on your filesystem, but that you have run out of SysV IPC semaphores.

You can see what your limits are like this:

# cat /proc/sys/kernel/sem
250 32000 32 128

I typically double those limits by adding this line to /etc/sysctl.conf:

kernel.sem = 500 64000 64 256

That makes sure you’ll get the change at the next boot. To make the change take immediate effect:

# sysctl -p

With those limits I’ve run 100 Apache instances on the same server.

DevCamps news

2011-08-04T00:00:00+00:00

DevCamps is a system for managing development, integration, staging, and production environments. It was developed by End Point for, and with the help of, some of our ecommerce clients. It grew over the space of several years, and really started to become its own standalone project in 2007.

Camps are a behind-the-scenes workhorse of our web application development at End Point, and don’t always get much attention because everyone’s too busy using camps to get work done! But this summer a few things are happening.

In early July we unveiled a redesign of the devcamps.org website that features a more whimsical look, a better explanation of what camps are all about, and endorsements by business and developer users. Marko Bijelic of Hipinspire did the design. Take a look:

www.devcamps.org

In less than two weeks, on August 17, I’m going to be giving a talk on camps at YAPC::EU in Riga, Latvia. YAPC::EU is Europe’s annual Perl conference, and will be a nice place to talk about camps.

Many Perl developers are doing web applications, which is camps’ main focus, so that’s reason enough. But camps also started around the Interchange application server, which is written in Perl. And the camp system is currently implemented in Perl as well.

We’ve set up a lot of camp systems for Perl web applications. So even though we’ve also set up camp systems for web applications using Ruby on Rails, Sinatra, Django, and PHP, it’s a nice homecoming to talk about camps to Perl enthusiasts.

Using nginx to transparently modify/debug third-party content

2011-02-06T00:00:00+00:00

In tracking down a recent front-end bug for one of our client sites, I found myself needing to use the browser’s JavaScript debugger for stepping through some JavaScript code that lived in a mix of domains; this included a third-party framework as well as locally-hosted code which interfaced with—and potentially interfered with—said third-party code. (We’ll call said code foo.min.js for the purposes of this article.) The third-party code was a feature that was integrated into the client site using a custom domain name and was hosted and controlled by the third-party service with no ability for us to change directly. The custom domain name was part of a chain of CNAMEs which eventually pointed to the underlying actual IP of the third-party service, so their infrastructure obviously relied on getting the Host header correctly in the request to select which among many clients was being served.

It appeared as if there was a conflict between code on our site and that imported by the third party service. As part of the debugging process, I was stepping through the JavaScript in order to determine what if any conflicts there were, as well as their nature (e.g., conflicting library definitions, etc.). Stepping through our code was fine, however the third-party’s JS code was (a) unfamiliar, and (b) minified, so this had the effect of putting all of the JavaScript code more-or-less on one line, which made tracing through the code in the debugger much less useful than I had hoped.

My first instinct was to use a JavaScript beautifier to reverse the minification process, but since I had no control over the code being included from the third-party service, this did not seem to be directly feasible. The third-party code was deployed only on our production site and relied on hard-coded domains which would make integrating it into one of our development instances challenging since we had no control over the contents of the returned resources. Since the relevant feature (and subsequent bugs) was only on the production site, making extensive modifications to how things were done and potentially breaking that or other features for users while I was debugging was obviously out as an option.

Enter nginx. I’ve been doing a lot with nginx lately as far as using it as a reverse proxy cache, so it’s been on my mind lately. So I came up with this technique:

Look up the IP address for the third-party’s domain name (used for later purposes).
Install nginx on localhost, listening to port 80.
Modify /etc/hosts to point the third-party’s domain name to the nginx server’s IP (also localhost in this case).

Configure a new virtual host with the following logical constraints:

 <ul>
   <li>We want to serve specific files (the beautified JavaScript) from our local server.
   <li>We want any other request going through that domain to be passed-through transparently, so neither the browser nor the third-party server treat it differently.
 </ul>

Given these constraints, this is the minimal configuration that I came up with (the interesting parts are located in the server block):

/etc/hosts:

example.domain.com 127.0.0.1

nginx.conf:

worker_processes 1;

events {
    worker_connections 10;
}

http {
    include       mime.types;
    default_type  application/octet-stream;
    
    server {
        server_name example.domain.com;
        root /path/to/local_root;

        try_files $uri @proxied;

        location @proxied {
            proxy_set_header Host $http_host;
            proxy_pass http://1.2.3.4;
        }
    }
}

Once I had the above configured/setup, I downloaded/saved the foo.min.js file from the third-party service, ran it through a JS beautifier, and saved it in the local nginx’s cache root so it would be served up instead of the actual file from the third-party service. Any other requests for static resources (images, other scripts, etc) would pass-through to the third-party server, so I had my nicely-formatted JavaScript code to step through, the production site worked as normal for anyone else despite potential local changes to the file on my end (i.e., adding JavaScript alert() calls to the file, and no one was the wiser.

A few notes

The try_files directive instructs nginx to first look for a file named after the current URI (foo.min.js in our example) in our local cache, and if this is not found, then fallback to the proxied location block; i.e., relay the request to the original upstream server. We explicitly set the Host header on the proxy request because we want the request to behave normally with respect to name-based hosting, and provide the saved IP address to contact the server in question.

We only needed to preserve/lookup the upstream server’s IP address because we’re running the nginx server on localhost, so if we used a domain name the lookup would return the same IP defined in /etc/hosts; if the nginx server was running on a different machine, you would be able to just use the domain name as both the server_name and the proxy_pass parameters and set the entry for the host in your local /etc/hosts file to the IP of the nginx server.

A possible extension would be to detect when an upstream request matched a minified URL (via a location ~ .min..*.js$ block) and automatically beautify/cache the content in our local cache. This could be accomplished via the use of an external FastCGI script to retrieve, post-process, and cache the content.

This technique can also be used when dealing with testing changes to a production site on which you are unable or unwilling to make potentially disruptive changes for the purposes of testing static resources. (JavaScript seems the most obvious application here, but this could apply to serving up images or other static content which would be resolvable by the local cache.)

I always need to remind myself to undo changes to /etc/hosts as soon as I’m done testing when using tricks like these. Particularly in something like this which is more-or-less transparent, the behavior would be functionaly identical as long as code/scripts on the third-party site stayed the same, but could easily introduce subtle bugs if the third-party services made changes to their codebase. Since our local copies would mask any remote changes for those non-proxied resources, this could be very confusing if you forget that things are set up this way.

DevCamps on different systems, including Plesk, CPanel and ISPConfig

2010-01-08T00:00:00+00:00

In the last few months I’ve been active setting up DevCamps for several of our newer clients. DevCamps is an open source development environment system, that once setup, allows for easily starting up and tearing down a development environment for a specific site/code base.

I’ve done many camps setups, and you tend to run into surprises from system to system, but what was most interesting and challenging about these latest installs was that they were to be done on systems running Plesk, CPanel, and ISPConfig. Some things that are different between a normal deployment and one on the above mentioned platforms are:

On the Plesk system there was a secured Linux called ‘Atomic Secured Linux’ which includes the grsecurity module. One restriction of this module is (TPE) Trusted Path Execution which required the camp bin scripts to be owned by root and the bin directory could not be writable by other groups, otherwise they would fail to run.
Permissions are a mixed bag, where typically we set all of the files to be owned by the site owner, in Plesk there are special groups such as psacln that the files need to be owned by.
On the CPanel system we needed to move the admin images for Interchange to a different directory since CPanel includes Interchange and has aliases for /interchange/ and /interchange-5/ to point at a central location which we would not be using.
On ISPConfig and Plesk the home directories of the sites are in different places, which required deploying the code in such places as /var/www/clients/client/user/domain.com or /var/www/vhosts/domain.com.

In the end we were able to get DevCamps to run properly on these various platforms both in development and production. If you are starting a new project or working on an existing project and could use a strong development environment, consider DevCamps.

Subverting Subversion for Fun and Profit

2009-04-07T00:00:00+00:00

One of our clients recently discovered a bug in a little-used but vital portion of the admin functionality of their site. (Stay with me here…) After traditional debugging techniques failed on the code in question, it was time to look to the VCS for identifying the regression.

We fortunately had the code for their site in version control, which is obviously a big win. Unfortunately (for me, at least), the repository was stored in Subversion, which means that my bag o’ tricks was significantly diminished compared to my favorite VCS, git. After attempting to use ‘svn log/svn diff -c’ to help identify the culprit based on what I thought the issue might be, I realized that svn was just not up to the task.

Enter git-svn. Using git svn clone file://path/to/repository/trunk, I was able to acquire a git-ized version of the application’s repository. For this client, we use DevCamps exclusively, so the entire application stack is stored in the local directory and run locally, including apache instance and postgres cluster. These pieces are necessarily unversioned, and are ignored in the repository setup. I was able to stop all camp services in the old camp directory (svn-based), rsync over all unversioned files to the new git repository (excluding the .svn metadata), replace the svn-based camp with the new git-svn based one, and fire up the camp services again. Started up immediately and worked like a charm. I now had git installed and working in what had previously only been svn-capable before.

Now that I had a git installation, I was able to pull one of my favorite tools from my toolbox when fighting regressions: git-bisect. In my previous svn contortions, I had located a previous revision several hundred commits back which did not exhibit the regression, so I was able to start the bisect with the following command: git bisect start bad good. In this case, bad was master and good was the revision I had found previously. Using git svn find-rev rnumber, I found the SHA1 commit for the good ref as git saw it.

From this point, I was able to quickly identify the commit which introduced the regression. In reviewing the diff, there was nothing that I would have expected to cause the issue at hand; the code did not touch any of the affected area of the admin. But git had never lied to me before. I compared the code currently in master with that introduced in the implicated commit and saw that most of it was still in place. I began selectively commenting out pieces of the code the commit introduced, and was able to enable/disable the bug with increasingly fine granularity. Finally, I was able to identify the single line which when removed caused the issue to evaporate. This was a line in an innocuous template which had a simple variable interpolation (inside an HTML comment, nonetheless); however, this line (which was in a file which was included with every document, added in the implicated commit) revealed a bug in the parser of the app-server which was causing the symptoms in the unrelated admin area.

It’s certain that I would never have been able to find the source of this issue without git-bisect, as manual bisection with svn would have been too tedious to even consider. I am able to happily interact with the rest of the development team with git being my secret weapon; git svn dcommit enables me to push my commits upstream, and git svn fetch/git svn rebase enable me to pull in the upstream changes. I’ll never need to tell my subversive secret (except, you know, on the company blog), and my own happiness and productivity has increased. Profit!!11 all around.

Camps presentation at UTOSC 2008

2008-08-31T00:00:00+00:00

Friday evening I did a presentation on and demonstration of our “development camps” at the 2008 Utah Open Source Conference in Salt Lake City. Attendees seemed to get what camps are all about, asked some good questions, and we had some good conversations afterwards. You can read my presentation abstract and my slides and notes, and more will be coming soon at the camps website.

I’ll post more later on some talks I attended and enjoyed at the conference.