Puppet, Salt, and DevOps (a review of the MountainWest DevOps conference)

2014-03-27T00:00:00+00:00

Last week I attended the MountainWest DevOps conference held in Salt Lake City, Utah. This was a one day conference with a good set of presenters and lightning talks. There were several interesting topics presented, but I’ll only review a few I wanted to highlight.

I Serve No Master!

Aaron Gibson of Adaptive Computing discussed a very common problem with Puppet (and other configuration management systems): they work well in the scenario they were designed for but what about when the situation isn’t typical? Aaron had a situation where developers and QA engineers could instantiate systems themselves via OpenStack, however the process for installing their company’s software stack on those VMs was inconsistent, mostly manual, and took many hours. One of the pain points he shared, which I related to, was dealing with registering a puppet node with a puppet master—the sometimes painful back and forth of certificate issuing and signing.

His solution was to remove the puppet master completely from the process. Instead he created a bash wrapper script to execute a workflow around what was needed, still using puppet manifests on each system but run locally. This wrapper tool, called “Builder”, relies on property files to customize the config and allow the script to manage different needs. This new script allowed them to keep using puppet to manage these self-serve OpenStack servers gaining the benefits of consistency and removing manual setup steps, providing the ability to automate installs with Jenkins or other tools. But it freed them from having to use a puppet master for nodes that were disposable. It also helped to reduce software install time from 12 hours down to a 11 minutes.

His Builder tool is still an internal only tool for his company, but he discussed some next steps he would like to add, including better reporting and auditing of executions. I pinged him after the conference on twitter and mentioned that Rundeck might be a good fit to fill that gap. I used Rundeck for 2 years at my last job, integrating nicely with other automation tools and providing reporting and auditing as well as access control of arbitrary jobs.

Automating cloud factories and the internet assembly line with SaltStack

Tom Hatch of Salt Stack spoke about Salt as an automation and remote execution platform. I’ve done quite a bit of work with Salt recently with a client and so I was pretty familiar with Salt. But one thing that he mentioned I didn’t know was that Salt was originally designed as a Cloud management tool, not necessarily a configuration management tool. However in the course of time configuration management became a higher priority for the Salt dev team to focus on. Tom mentioned that recently they have been working on Cloud management tools again—providing integration with Rackspace, AWS, xen, and more. I’ll have to dig more into these tools and give them a try.

How I Learned to Stop Worrying and Love DevOps

Bridget Kromhout of 8thBridge (since aquired by Fluid) spoke on the culture of DevOps and her journey from a corporate, strictly siloed environment to a small start-up that embraced DevOps. One of the first things she brought up that was different was the focus and approach to goals of each organization. In an organization where Ops teams are strictly separate from Developers, they often butt heads and have a limited vision of priorities. Each focuses on the goals of their own team or department, and have little understanding of the goals of the other departments. This leads to an adversarial relationship and culture of not caring much about different teams or departments.

In contrast, the organization that embraces DevOps as a culture will see to it that Ops and Devs work together on whatever solution best reaches the goals of the whole organization. In doing so, barriers will have to be questioned. Any “special snowflake” servers/applications/etc. that only one person knows and can touch can’t exist in this culture. Instead, any unique customizations need to be minimized through automation, documentation (sharing knowledge), and reporting/monitoring. This doesn’t mean root access for all—but it means reducing barriers as much as possible. Good habits from the Ops world to keep include: monitoring, robustness, security, scaling, and alerting.

The main pillars of DevOps are: culture, automation, measurement, and sharing. Culture is important and can be supportive or rejecting of the other pillars. Without a culture in the organization that supports DevOps, it will fizzle back into siloed “us vs. them” enmity.

Thanks to all the presenters and those that put on the conference. It was a great experience and I am glad I attended.

Setting a server role in Salt (comparing Puppet and Salt)

2013-12-23T00:00:00+00:00

There are many ways to solve a given problem, and this is no truer than with configuration management. Salt (https://saltstack.com) is a fairly new tool in the configuration management arena joining the ranks of Puppet, Chef, and others. It has quickly gained and continues to grow in popularity, boasting its scalable architecture and speed. And so with multiple tools and multiple ways to use each tool, it can get a little tricky to know how best to solve your problem.

Recently I’ve been working with a client to convert their configuration management from Puppet to Salt. This involved reviewing their Puppet configs and designs and more-or-less mapping them to the equivalent for Salt. Most features do convert pretty easily. However, we did run into something that didn’t at first/assigning a role to a server.

We wanted to preserve the “feeling” of the configs where possible. In Puppet they had developed and used a convention for using some custom variables in their configs to assign an “environment” and a “role” for each server. These variables were assigned in the node and role manifests. But in Salt we struggled to find a similar way to do that, but here is what we learned.

In Puppet, once a server’s “role” and “environment” variables were set, then they could be used in other manifest files to select the proper source for a given config file like so:

    file    {
        "/etc/rsyslog.conf":
            source  =>
            [
                "puppet:///rsyslog/rsyslog.conf.$hostname",
                "puppet:///rsyslog/rsyslog.conf.$system_environment-$system_role",
                "puppet:///rsyslog/rsyslog.conf.$system_role",
                "puppet:///rsyslog/rsyslog.conf.$system_environment",
                "puppet:///rsyslog/rsyslog.conf"
            ],
            ensure  => present,
            owner   => "root",
            group   => "root",
            mode    => "644"
    }

Puppet will search the list of source files in order and use the first one that exists. For example, if $hostname = ‘myniftyhostname’ and $system_environment = ‘qa’ and $system_role = ‘sessiondb’, then it will use rsyslog.conf.myniftyhostname if it exists on the Puppet master, or if not then use rsyslog.conf.qa-sessiondb if it exists, or if not then rsyslog.conf.sessiondb if it exists, or if not then rsyslog.conf.qa if it exists, or if not then rsyslog.conf.

In Salt, environment is built into the top.sls file, where you match your servers to their respective state file(s), and can be used within state files as {{ env }}. Salt also allows for multiple sources for a managed file to be listed in order and it will use the first one that exists in the same way as Puppet. We were nearly there; however, setting the server role variable was not as straight forward in Salt.

We first looked at using Jinja variables (which is the default templating system for Salt), but soon found that setting a Jinja variable in one state file does not carry over to another state file. Jinja variables remain only in the scope of the file they were created in, at least in Salt.

The next thing we looked at was using Pillar, which is a way to set custom variables from the Salt master to given hosts (or minions). Pillar uses a structure very similar to Salt’s top.sls structure- matching a host with its state files. But since the hostnames for this client vary considerably and don’t lend themselves to pattern matching easily, this would be cumbersome to manage both the state top.sls file and the Pillar top.sls file and keep them in sync. It would require basically duplicating the list of hosts in two files, which could get out of sync over time.

We asked the salt community on #salt on Freenode.net how they might solve this problem, and the recommended answer was to set a custom grain. Grains are a set of properties for a given host, collected from the host itself- such as, hostname, cpu architecture, cpu model, kernel version, total ram, etc. There are multiple ways to set custom grains, but after some digging we found how to set them from within a state file. This meant that we could do something like this in a “role” state file:

# sessiondb role
# {{ salt['grains.setval']('server_role','sessiondb') }}

include:
  - common
  - postgres

And then within the common/init.sls and postgres/init.sls state files we could use that server_role custom grain in selecting the right source file, like this:

/etc/rsyslog.conf:
  file.managed:
    - source:
      - salt://rsyslog/files/rsyslog.conf.{{ grains['host'] }}
      - salt://rsyslog/files/rsyslog.conf.{{ env }}-{{ grains['server_role'] }}
      - salt://rsyslog/files/rsyslog.conf.{{ grains['server_role'] }}
      - salt://rsyslog/files/rsyslog.conf.{{ env }}
      - salt://rsyslog/files/rsyslog.conf
    - mode: 644
    - user: root
    - group: root

This got us to our desired config structure. But like I said earlier, there are probably many ways to handle this type of problem. This may not even be the best way to handle server roles and environments in Salt, if we were more willing to change the “feeling” of the configs. But given the requirements and feedback form our client, this worked fine.

Puppet, Salt, and DevOps (a review of the MountainWest DevOps conference)

I Serve No Master!

Automating cloud factories and the internet assembly line with SaltStack

How I Learned to Stop Worrying and Love DevOps

Links

Setting a server role in Salt (comparing Puppet and Salt)