Puppet, Salt, and DevOps (a review of the MountainWest DevOps conference)

2014-03-27T00:00:00+00:00

Last week I attended the MountainWest DevOps conference held in Salt Lake City, Utah. This was a one day conference with a good set of presenters and lightning talks. There were several interesting topics presented, but I’ll only review a few I wanted to highlight.

I Serve No Master!

Aaron Gibson of Adaptive Computing discussed a very common problem with Puppet (and other configuration management systems): they work well in the scenario they were designed for but what about when the situation isn’t typical? Aaron had a situation where developers and QA engineers could instantiate systems themselves via OpenStack, however the process for installing their company’s software stack on those VMs was inconsistent, mostly manual, and took many hours. One of the pain points he shared, which I related to, was dealing with registering a puppet node with a puppet master—the sometimes painful back and forth of certificate issuing and signing.

His solution was to remove the puppet master completely from the process. Instead he created a bash wrapper script to execute a workflow around what was needed, still using puppet manifests on each system but run locally. This wrapper tool, called “Builder”, relies on property files to customize the config and allow the script to manage different needs. This new script allowed them to keep using puppet to manage these self-serve OpenStack servers gaining the benefits of consistency and removing manual setup steps, providing the ability to automate installs with Jenkins or other tools. But it freed them from having to use a puppet master for nodes that were disposable. It also helped to reduce software install time from 12 hours down to a 11 minutes.

His Builder tool is still an internal only tool for his company, but he discussed some next steps he would like to add, including better reporting and auditing of executions. I pinged him after the conference on twitter and mentioned that Rundeck might be a good fit to fill that gap. I used Rundeck for 2 years at my last job, integrating nicely with other automation tools and providing reporting and auditing as well as access control of arbitrary jobs.

Automating cloud factories and the internet assembly line with SaltStack

Tom Hatch of Salt Stack spoke about Salt as an automation and remote execution platform. I’ve done quite a bit of work with Salt recently with a client and so I was pretty familiar with Salt. But one thing that he mentioned I didn’t know was that Salt was originally designed as a Cloud management tool, not necessarily a configuration management tool. However in the course of time configuration management became a higher priority for the Salt dev team to focus on. Tom mentioned that recently they have been working on Cloud management tools again—providing integration with Rackspace, AWS, xen, and more. I’ll have to dig more into these tools and give them a try.

How I Learned to Stop Worrying and Love DevOps

Bridget Kromhout of 8thBridge (since aquired by Fluid) spoke on the culture of DevOps and her journey from a corporate, strictly siloed environment to a small start-up that embraced DevOps. One of the first things she brought up that was different was the focus and approach to goals of each organization. In an organization where Ops teams are strictly separate from Developers, they often butt heads and have a limited vision of priorities. Each focuses on the goals of their own team or department, and have little understanding of the goals of the other departments. This leads to an adversarial relationship and culture of not caring much about different teams or departments.

In contrast, the organization that embraces DevOps as a culture will see to it that Ops and Devs work together on whatever solution best reaches the goals of the whole organization. In doing so, barriers will have to be questioned. Any “special snowflake” servers/applications/etc. that only one person knows and can touch can’t exist in this culture. Instead, any unique customizations need to be minimized through automation, documentation (sharing knowledge), and reporting/monitoring. This doesn’t mean root access for all—but it means reducing barriers as much as possible. Good habits from the Ops world to keep include: monitoring, robustness, security, scaling, and alerting.

The main pillars of DevOps are: culture, automation, measurement, and sharing. Culture is important and can be supportive or rejecting of the other pillars. Without a culture in the organization that supports DevOps, it will fizzle back into siloed “us vs. them” enmity.

Thanks to all the presenters and those that put on the conference. It was a great experience and I am glad I attended.

Setting a server role in Salt (comparing Puppet and Salt)

2013-12-23T00:00:00+00:00

There are many ways to solve a given problem, and this is no truer than with configuration management. Salt (https://saltstack.com) is a fairly new tool in the configuration management arena joining the ranks of Puppet, Chef, and others. It has quickly gained and continues to grow in popularity, boasting its scalable architecture and speed. And so with multiple tools and multiple ways to use each tool, it can get a little tricky to know how best to solve your problem.

Recently I’ve been working with a client to convert their configuration management from Puppet to Salt. This involved reviewing their Puppet configs and designs and more-or-less mapping them to the equivalent for Salt. Most features do convert pretty easily. However, we did run into something that didn’t at first/assigning a role to a server.

We wanted to preserve the “feeling” of the configs where possible. In Puppet they had developed and used a convention for using some custom variables in their configs to assign an “environment” and a “role” for each server. These variables were assigned in the node and role manifests. But in Salt we struggled to find a similar way to do that, but here is what we learned.

In Puppet, once a server’s “role” and “environment” variables were set, then they could be used in other manifest files to select the proper source for a given config file like so:

    file    {
        "/etc/rsyslog.conf":
            source  =>
            [
                "puppet:///rsyslog/rsyslog.conf.$hostname",
                "puppet:///rsyslog/rsyslog.conf.$system_environment-$system_role",
                "puppet:///rsyslog/rsyslog.conf.$system_role",
                "puppet:///rsyslog/rsyslog.conf.$system_environment",
                "puppet:///rsyslog/rsyslog.conf"
            ],
            ensure  => present,
            owner   => "root",
            group   => "root",
            mode    => "644"
    }

Puppet will search the list of source files in order and use the first one that exists. For example, if $hostname = ‘myniftyhostname’ and $system_environment = ‘qa’ and $system_role = ‘sessiondb’, then it will use rsyslog.conf.myniftyhostname if it exists on the Puppet master, or if not then use rsyslog.conf.qa-sessiondb if it exists, or if not then rsyslog.conf.sessiondb if it exists, or if not then rsyslog.conf.qa if it exists, or if not then rsyslog.conf.

In Salt, environment is built into the top.sls file, where you match your servers to their respective state file(s), and can be used within state files as {{ env }}. Salt also allows for multiple sources for a managed file to be listed in order and it will use the first one that exists in the same way as Puppet. We were nearly there; however, setting the server role variable was not as straight forward in Salt.

We first looked at using Jinja variables (which is the default templating system for Salt), but soon found that setting a Jinja variable in one state file does not carry over to another state file. Jinja variables remain only in the scope of the file they were created in, at least in Salt.

The next thing we looked at was using Pillar, which is a way to set custom variables from the Salt master to given hosts (or minions). Pillar uses a structure very similar to Salt’s top.sls structure- matching a host with its state files. But since the hostnames for this client vary considerably and don’t lend themselves to pattern matching easily, this would be cumbersome to manage both the state top.sls file and the Pillar top.sls file and keep them in sync. It would require basically duplicating the list of hosts in two files, which could get out of sync over time.

We asked the salt community on #salt on Freenode.net how they might solve this problem, and the recommended answer was to set a custom grain. Grains are a set of properties for a given host, collected from the host itself- such as, hostname, cpu architecture, cpu model, kernel version, total ram, etc. There are multiple ways to set custom grains, but after some digging we found how to set them from within a state file. This meant that we could do something like this in a “role” state file:

# sessiondb role
# {{ salt['grains.setval']('server_role','sessiondb') }}

include:
  - common
  - postgres

And then within the common/init.sls and postgres/init.sls state files we could use that server_role custom grain in selecting the right source file, like this:

/etc/rsyslog.conf:
  file.managed:
    - source:
      - salt://rsyslog/files/rsyslog.conf.{{ grains['host'] }}
      - salt://rsyslog/files/rsyslog.conf.{{ env }}-{{ grains['server_role'] }}
      - salt://rsyslog/files/rsyslog.conf.{{ grains['server_role'] }}
      - salt://rsyslog/files/rsyslog.conf.{{ env }}
      - salt://rsyslog/files/rsyslog.conf
    - mode: 644
    - user: root
    - group: root

This got us to our desired config structure. But like I said earlier, there are probably many ways to handle this type of problem. This may not even be the best way to handle server roles and environments in Salt, if we were more willing to change the “feeling” of the configs. But given the requirements and feedback form our client, this worked fine.

Chef and Puppet Overview

2012-06-13T00:00:00+00:00

I started a job several years ago as a “configuration manager”, but had to admit when I started that I didn’t have any idea what “configuration management” really meant. The idea, as I soon learned, was to make sure all the servers, configurations, accounts, and other components of a system work together systematically. I’m not sure “configuration management” tools as such existed at the time, but we certainly never used them, though they’d begun to have a presence online before leaving that job for another opportunity.

In systems we run at End Point, whether for ourselves or other clients, such configuration management tools have become critical, in particular for our Liquid Galaxy systems, which require a great deal of repetitive configuration. So Kiel and Josh Williams have a fair bit of experience with these tools, and I was glad to hear their discussion of Chef and Puppet specifically.

These tools have a common heritage, and are both Ruby-based. Ruby is a particularly good language for writing domain-specific languages (DSLs) like the one Puppet uses, so it’s interesting that Chef’s developers apparently abandoned the DSL idea, so Chef instructs its users run Ruby directly. Chef is newer, spawned by dissatisfied Puppet developers and users. We’re generally shifting toward Chef after concluding we share many of those dissatisfactions, but both have proven very time saving, for us.

Kiel told of one client whose dozens of app servers we rebuilt in a day, essentially one at a time, simply by kicking off Puppet tasks on each one in turn. As mentioned above, the Liquid Galaxy uses Chef as well. Whereas it used to take a fully day of manual work building what’s called the Galaxy’s “head node”, now, in combination with scripted, automatic operating system installations, we can set up a new head node in minutes. We’re still working to get everything into Chef, but in particular all the monitoring scripts and tools used on a Liquid Galaxy are built from Chef recipes, so every new head node is all ready to monitor from the beginning. Since we deploy these systems all over the world, and must manage them remotely, this is critically important.

Building “recipes” for these tools—that is, sets of instructions that tell the tool what to build—can be a detailed and difficult process. Kiel and Josh recommended being explicit about the configuration you actually want, rather than simply accepting defaults, principally because later on, it’s difficult to know precisely what the original author intended. They also recommend starting with small, easily tested services, such as NTP. For many systems, breaking NTP for a while won’t cause problems, so it can be a good service to begin playing with.

One attendee was curious to know how many servers a system needed to involve for Chef or Puppet to be worth considering. The rule of thumb is, apparently, about 10, but Josh Williams suggested having “more than one” was enough to start writing recipes. I guess I’m sold.

Puppet PDX meeting

2009-02-19T00:00:00+00:00

Even zombies like puppets!

If you live in or near Portland, OR, come join us for our first Puppet PDX meeting. We’re meeting from 6-7pm, at Paddy’s on SW 1st and Yamhill in Downtown Portland. It’s right on the MAX line.

Inspired by Puppet and the Reductive Labs team, we’re gathering people interested in all things related to configuration management. (Not sock puppet-making, sorry!) Cfengine user? Thinking about trying AutomateIt? Just have a pile of obsessively managed scripts? Come on down! We’ll discuss tools, best practices, and generally how to make your systems run so well you can get to the pub by 4 o’clock.

This is the first meetup. Hopefully we can get organized, get to know each other, and decide on what the goals of the group should be.

So if you are interested in automation, configuration management, cloud computing, and large scale computing environments, come join us for a few drinks and some lively chatter.

Please RSVP by sending an email to puppetpdx@reductivelabs.com so that we can get a bigger room if needed.