Blog

A server build checklist – that checks itself!

The process of building a server – physical or virtual – has so many steps that we write checklists to make sure we didn’t miss any.  While they’re valuable to have, they cause extra work to check on each new system, and may not get run at all on cloud servers that spin up in response to load.

Can’t we have a checklist that automatically performs the checks on each new server without any human effort?

Server customization

Let’s take a look at some sample tasks that have to be performed on every new web server.

  • Install httpd and mod_ssl packages
  • Remove the gcc compiler (since we don’t want to compile code on production machines, and many strains of malware need a compiler to propagate)
  • Download the SSL key and certificate and check that the key is only readable by user httpd
  • Create local user accounts for jparker and cadams, our two administrators
  • Make sure we’re sending all logs to 2 central syslog servers

Obviously there are many more, but this makes a usable sample for our demonstration.

Get 10 DevOps in a room and you’ll hear 10 different ways to do these customizations:  chef or puppet recipes, kickstart tasks done at the end of the OS install, customizations performed once on a gold master that’s cloned to make the real web servers, a shell script that’s downloaded and run, or customizations done by hand at the command line.  The problem shared by all the install methods is that they can fail. Some different ways they could fail…

  • During the install, a git server that holds the key and certificate is being rebooted, so we never download those.
  • A network link drops, so we never get the httpd or mod_ssl packages
  • A new DevOps is asked to create a new web server and isn’t aware of the additional customizations needed.
  • Someone’s rushed or distracted and forgets to perform one of the steps.
  • Gold images move quickly out of date and don’t get updated.

What we need is an automatic way to verify that all the customizations were done to prevent errors and missteps from happening.

Halo Configuration Policies

Many Halo users already use the prebuilt policies we’ve provided for the base operating system or server applications.  These policies check for common security risks and create a report or send alerts so you can remediate them.

You can create your own configuration policies from scratch and we’ll do that here to show you how to check that all your configuration steps have been performed.  If one was missed, you’ll see a failure in the your server scan report (server scans run daily; you can also run on-demand manual scans to check as you go).

First, create a configuration policy called “Apache server checklist” and apply it to your “Web Servers” group:

server build checklist

Now open the policy up for editing.  Let’s look at the individual items in our sample checklist and see how they could be checked in our new policy.

Rule: Install httpd and mod_ssl packages

Checklist:

  1. Install httpd and mod_ssl packages
  2. Remove the gcc compiler
  3. Download the SSL key and certificate and check that the key is only readable by user httpd
  4. Create local user accounts for jparker and cadams, our two administrators
  5. Make sure we’re sending all logs to 2 central syslog servers

This and the next 2 rules will go under “Software configuration” in our policy.

We can’t directly test for installed packages.  To test whether a package is installed or not we can see if a file from that package exists on the system.  For httpd, we want to test that /usr/sbin/httpd exists on the system.  For mod_ssl, we’ll check that /etc/httpd/conf.d/ssl.conf is there.  This File Presence check will tell us that those two packages are installed:

server build checklist

To pick a flag file like /usr/sbin/httpd for a given package, look at the package file listing.  For an installed package, use:

rpm -ql httpd | less

For a package that’s not installed, download one of the rpms for that package and run:

rpm -qpl httpd-2.2.22-4.fc17.x86_64.rpm | less

,substituting the name of the rpm file.  Any of the files you find should work fine as a flag file.

Rule: gcc package is removed

Checklist:

  1. Install httpd and mod_ssl packages
  2. Remove the gcc compiler
  3. Download the SSL key and certificate and check that the key is only readable by user httpd
  4. Create local user accounts for jparker and cadams, our two administrators
  5. Make sure we’re sending all logs to 2 central syslog servers

Like above, we’ll put in a File Presence check, but we want to check that a file is absent rather than present.  We’ll check for the absence of /usr/bin/gcc for the gcc package.

server build checklist

Rule: Install web server SSL key and certificate and check permissions

Checklist:

  1. Install httpd and mod_ssl packages
  2. Remove the gcc compiler
  3. Download the SSL key and certificate and check that the key is only readable by user httpd
  4. Create local user accounts for jparker and cadams, our two administrators
  5. Make sure we’re sending all logs to 2 central syslog servers

 

First, we want to make sure that /etc/httpd/conf/ssl.key/server.key and /etc/httpd/conf/ssl.crt/server.crt exist with a File Presence check like above.

server build checklist

Because the SSL key is very sensitive, we also want to use a File Ownership check to make sure it’s owned by user httpd, and a File ACL check to make sure it’s not world readable.  We actually check to make sure that the file is not readable, writable, or executable by anyone other than its owners.

server build checklist

We don’t care about the first two digits of the file ACL as these respectively deal with user owner and group owner permissions.  We do care about the third digit, though, as this may grant permissions to all other users on the system.  We want that to be a 0 to block all other users from reading, writing (or executing, although that has no effect here) that file.

The Configuration Policy system doesn’t have an explicit check to force a “0” in the third position; we use a slightly different syntax with the same effect.  We use “NOT: **1,**2,**3,**4,**5,**6,**7” to declare that the third digit cannot be any value between 1 and 7.  Since 0-7 are the only possible values, this has the end effect to alerting if the file has any permissions for all non-owners.

Rule: Admin accounts created

Checklist:

  1. Install httpd and mod_ssl packages
  2. Remove the gcc compiler
  3. Download the SSL key and certificate and check that the key is only readable by user httpd
  4. Create local user accounts for jparker and cadams, our two administrators
  5. Make sure we’re sending all logs to 2 central syslog servers

These last 2 rules will go under “System configuration” in the policy.

There are two ways to check that a particular user account is created with Halo; check that there’s an “{account_name}:….” line in /etc/passwd, or look for a directory called “/home/{account_name}”.  The latter is not as good a choice because a user account might have their home directory in a different place, so we’ll check /etc/passwd instead.

We can use String Presence checks to look for
^jparker:
and
^cadams:
in /etc/passwd .

server build checklist

server build checklist

Rule: Syslog sends logs to central log servers

Checklist:

  1. Install httpd and mod_ssl packages
  2. Remove the gcc compiler
  3. Download the SSL key and certificate and check that the key is only readable by user httpd
  4. Create local user accounts for jparker and cadams, our two administrators
  5. Make sure we’re sending all logs to 2 central syslog servers

We want /etc/syslog.conf to have one line per central syslog server, which will look like:

*.* @1.2.3.4

Here’s an example that checks that we have both 1.2.3.4 and 1.2.3.75 as central syslog servers:

server build checklist

server build checklist

You’ll have one check for each central syslog server.

Layering Configuration Policies

You can apply multiple configuration policies to your servers as needed – if you remember at the beginning of this process, we applied our “Apache Server Checklist” to this group that already had a core security policy applied (“CentOS and RHEL Linux core policy v. 2”), so Halo is scanning for the checks in both policies.  We can use that layering capability in writing our checklist policy as well – for example, only two of the above checks were specific to Apache (“Install httpd and mod_ssl” and “Install certificate and check permissions”). The other three checks could very reasonably be in a “General server checklist” policy, leaving the two apache checks in “Apache server checklist”.  We can apply both policies to the “Web server” group, and apply the “General server checklist” to all the other server groups.  In short, set up a “General” policy for checks that apply to all your servers, and Application/Task/Group Specific policies as needed.

Using this method, your CentOS Linux web server group can very reasonably have three configuration policies: the CentOS Linux distribution policy, the Apache configuration policy, and the Global settings configuration policy.

Stay up to date

Get the latest news and tips on protecting critical business assets.

Related Posts