Don’t script it, just make it so

Matt Simmons recently had another post that mentioned structured systems management. I find myself re-reading Ad-Hoc vs Structured Systems Management from time to time and I figure its time for me to chime in.

First off, Michael Jankes’ post is one of my favorite blog post reads. Few posts do I ever re-read unless its tutorialesque. For a quick sidetrack one of those posts that I do return to when I deal with developers is The Joel Test: 12 Steps to Better Code. Back on subject …

One of the most frequent things that comes up when people start talking about good administration practices is scripting things. If you can’t script it make a checklist is a common sentiment. It’s not that I don’t think that scripting things is a good idea because I do. I would go so far to say that scripting things is an important part of documentation. You should be able to reproduce a system from documentation. Any configuration that can be done programatically should be and it should be in your documentation so in case of catastrophic failure someone can reproduce your work relatively easily. If I like scripting so much why is this post titled “Don’t script it”?

I think there is a better way. Use a configuration management system. Declare your configuration and allow the configuration management system to make it happen. This has many advantages over simply scripting something. Configuration can be enforced over time. I know I have been guilty of making small changes to configurations to see how things go. While those small changes seem to trickle into production systems they never (or at least rarely) seem to trickle into the documentation. Before you know it you have a pile of undocumented small changes that really encompass an entirely different configuration from what you started with.

A well placed configuration management system helps avoid these issues. Store your configuration in version control. I might even go so far as to say to automate the update of the checkout your configuration management server uses so that you don’t allow yourself to make small changes to the management system configuration without checking it into your version control. A few tweaks being automatically removed will go a long way to drive home the point that your changes need to be in version control. Placing your config in version control also allows you to see your changes over time. You can look back through the log and your comments to see when and why changes were made. This can be especially important if you are not the only person making changes to configuration (ah hem Matt, I know your hiring someone you better version your config before your no longer the lone admin).

This all comes up because I have been doing more work with puppet recently. It takes a while to wrap your head around but I don’t really think I have heard any arguments against that type of system being the “right” way to do systems administration. If I have they sure didn’t register. I like puppet because it works on a variety of systems and abstracts things like package management. I just tell my configuration system to make sure nodeX has package x installed and the puppet client figures out if its RH or Debian or SUSE or Solaris or whatever and runs the proper commands to get the package installed.

I am interested in spacewalk as well. But right now it only runs on Oracle databases so until it runs on postgresql there really isn’t even a point in me looking as the places I would be able to use it would be very limited.