Blog

Building a Firewall: Part 3

If you were given a computer and asked to build a firewall for it, how would you find the services and ports in the first place?

To recap, in part 1 we covered what a firewall is and how it works with the bouncer-in-a-club analogy.  In part 2, we actually loaded this into Halo to set up a live firewall for a LAMP server.  Because those two blogs were focused on the firewall process, they quietly skipped over the job of deciding what ports need to be opened.

Finding the services and ports to cover in the firewall

To help simplify the process of writing your first firewall in the previous blogs, I provided you with a list of services and the ports they used.  Let’s step back now and see how that’s done.

Port Numbers

First, you can find a list of commonly used port numbers and their names in “/etc/services” on Linux, Unix, BSD, and MacOS systems.  In Windows this can be found in “C:Windowssystem32driversetcservices”.  This file won’t hold every possible service name and service port number combination, but holds the more commonly used ones.

If you come across a port number you don’t recognize, you can fall back on the database of ports used by experienced firewall designers – do a Google search for “TCP port {port number}”.  🙂

Netstat – what’s open on my machine

Linux, MacOS, and Windows all provide a tool called netstat that shows:

On which ports your machine is listening
What connections have been made between 2 computers.

Both of these are very helpful in the process, so let’s take a look.  On Linux, either run this command as the root user:

netstat -anp | less -S

 

or this command as a non-root user:

sudo netstat -anp | less -S

 

On a Windows system, you can produce similar output with the command:

netstat -aonb | more

 

With either form, you’ll see output like this:

I wouldn’t fault you if you felt that was as readable as ancient Babylonian, but bear with me.  🙂  I’ll show you where to look.

We’re interested in services that can be seen from the Internet and live connections to and from other machines.  All of those can be found in lines that start with either “tcp” or “udp”, so we can completely ignore everything from “Active UNIX domain sockets” down.

Netstat – TCP Servers

Next, let’s focus on the “tcp” lines, since most of what we want to share on the Internet is carried by TCP:

All the lines with the word “LISTEN” over to the right show servers; programs that wait for incoming connections.  For example, our web server (on the next to last line above) listens on tcp port 80 (see the port number in the 4th column to the right of the last “:”).  The Incoming chain of your firewall protects these servers – we’re putting barriers up in front of the ports we don’t want outsiders to reach.

On this machine, it looks like we have 6 LISTEN lines, used by the following programs: sshd, cupsd, sendmail, and httpd.

httpd

You’ve already seen httpd – it’s the Apache web server.  If you remember our original firewall, though, we opened up both port 80 (unencrypted HTTP) and port 443 (HTTP Secure, called HTTPS); on this system, Apache is only listening on port 80.  It’s not listening on port 443 because we haven’t configured any websites that use secure communications.

In a case like this where you know a program may listen on more than one port at some point and you know those ports in advance, it makes sense to configure the firewall for both ports now, as we did in the previous blogs.  If we allow port 443 through the firewall but there’s no server listening on that port, a client trying to connect to it will just get a “closed port” error.

There’s a slight risk in opening a port in advance like this.  Once the firewall allows in port 443 traffic, someone else on the system could start a different server listening on that port and the firewall would allow the traffic.  The firewall itself doesn’t know what program answers an incoming request – it just cares about packets and ports.  If this is a concern, Halo does have the ability to check this; you can make a “Network Service Processes” check that only allows the “httpd” process to be listening on port 443.

sshd

Next we have sshd, the ssh daemon (“daemon” is a synonym for server in this context):

Here’s another quirk – in our original firewall, sshd was listening on the standard port 22.  Why is this one listening on port 7722?

The person that runs this computer knows that there are automated scanners that go from machine to machine through the Internet looking for SSH servers.  When they find one, they send common usernames and passwords at the server, hoping to get lucky and break in because someone picked a weak password.  There are lots of defenses to this, including picking strong passwords and using a firewall to limit who can talk to the port, but this administrator is using one more technique – moving the SSH server to a different port.

When the scanner arrives at this machine, it will attempt to connect to port 22 and the answer they’ll get back is “sorry, that port is closed”.  Since the scanner wants to quickly find easy to guess passwords on easy to find SSH daemons, it will never look for an SSH server tucked away on a different port (in this case, 7722).

While you’re making a firewall, pay attention to this port field (the number after the colon in column 4).  Don’t just assume that because someone’s running sshd that it must be listening on port 22 – as we’ve seen, it can move to any of the 65,535 possible TCP ports.  Moving it means more effort – we have to tell the clients to use the alternate port – but it’s surprisingly effective at stopping scanners from guessing passwords.

A little further down the netstat output there’s one more sshd line:

The line is almost identical to the one above, except the first line used “0.0.0.0” in the “Local Address” and “Foreign Address” columns, and this line uses “::” in both (both lines use one more colon in front of the port number).  The first line is using IP version 4 addresses, while the second is using IP version 6 addresses.  As far as the firewall is concerned, there’s not much to note; the SSH daemon is ready to accept connections from both IPv4 and IPv6 clients, and it’s listening on port 7722 for both.

This raises a side issue: IPv6 and firewalls.  Not every firewall handles IPv6, and even those that do may require you to set up essentially a parallel firewall for IPv6 traffic.  If your firewall does not correctly handle IPv6 traffic or you’re not sure, the safest step is to disable IPv6 support on this machine entirely until that’s fixed.  The downside to this choice is that you will not be able to communicate with IPv6-only clients.

sendmail

Now let’s look at the sendmail program:

Sendmail is a common program used to carry mail throughout the internet, so this machine must be a mail server, right?

It’s a good guess, but there’s a critical difference between the lines we’ve seen above and this one – the address in the 4th column.  When Apache and sshd asked to listen on ports 80 and 7722 respectively, they didn’t put any restrictions on the clients who could connect – that’s why we see “0.0.0.0” in their 4th column; they’ll talk to anyone.  The sendmail program on this machine wants to accept connections on port 25 to accept incoming SMTP mail messages, but it has a restriction.  It asked to listen on port 25, but only on the loopback address 127.0.0.1.  If another program running on this machine connects to port 25, sendmail will accept the message and move it along.  If a program on the Internet comes in to talk to port 25, they get a “port closed” response.  What sendmail is doing is very much like a coarse firewall – it restricts who can talk to that port.

Even though sendmail is imposing some restrictions of it’s own, it’s still a good idea to decide whether this machine is supposed to be a mail server or not and put in an explicit firewall rule ALLOW’ing or DROP’ing port 25 traffic.  This protects the mail server software even if someone misconfigures it in the future and it starts listening for connections from anywhere.  This approach is called Defense In Depth – multiple layers of protection that can act as backups for each other – and it’s far better to do that in the end.

cupsd

The print server, cupsd (the Common Unix Printing System), has two lines:


The first line should look familiar; like sendmail above, cups only wants to accept connections to port 631 if they come from localhost.  The second line is identical, except it’s using the IPv6 version of localhost, “::1”.

Just like sendmail, the most secure approach is to explicitly put in a firewall rule for port 631.

That’s it for the tcp LISTEN’ing ports – *phew*!  Looking through lines of netstat output might not seem like fun, but it has one benefit – you get a very explicit listing of every listening port, so you know you haven’t missed anything in the end.

Netstat – TCP connections

OK, just two more lines to finish up TCP:

Neither line has “LISTEN” in the 6th column, so neither is a listening daemon.  The first is an open connection to a remote server.  In the 4th column (titled “Local Address”), we have 172.27.1.152:45438, and in the 5th column (“Foreign Address”) we have 172.27.1.182:22.  The foreign port looks like a standard ssh port, and the program holding it open (the last thing on the line) is, sure enough, “ssh”.  This is an SSH client making an outbound connection to the remote host 172.27.1.182 on the default SSH port of 22.  The connection uses a random high port, 45438, at this end, but we don’t generally focus on these client ports in writing firewall rules as they change with every single connection.

Would we use this in building a firewall?  Actually, yes.  Some of the ESTABLISHED lines you’ll see are just people on the outside world coming in to our services, and these connection lines are handled by the rules we already put in for servers likes httpd, sshd, mysql, etc.  But some of them are for outbound connections like the one we just analyzed.  Incoming ESTABLISHED lines will have the server port in the “Local Address” column, and Outgoing ESTABLISHED lines have the server port in the “Foreign Address” column (like the “:22” above, making this an outgoing connection).

When you’re putting together the “Outgoing” chain of rules, make sure you allow for Outbound traffic like Outbound ssh, or you won’t be able to make those connections any more.

So what’s the “TIME_WAIT” line?  It looks a lot like our ESTABLISHED line, except the local port is one smaller.  It was once an ESTABLISHED connection, but the connection closed up.  The operating system remembers it for a short time before completely forgetting it.  if you come back in two minutes that line will be gone.

When you see lines with some form of “_WAIT”, think of them just like ESTABLISHED lines; you need to make sure you have firewall rules to match them too.

Other outbound TCP

We were lucky that we happened to run netstat while an outbound SSH connection was live – that told us we need to include ssh in our Outgoing firewall rules.  Are there other types of outbound connections that we’re not seeing?

Probably yes.  Here’s where you have to think about what needs to happen on this system.

  • Essentially all machines make DNS lookups.  These might happen because we’re about to connect to an external server to check for software updates, or because the logging system wants to include hostnames in log files instead of just IP addresses.  If your firewall doesn’t automatically add rules to allow outbound DNS traffic, make sure you allow it.  That needs to include both outbound UDP port 53 and TCP port 53 traffic.
  • How does this computer get software updates?  Is this over HTTP, HTTPS, or perhaps a file share?
  • When people come in over SSH, is there any chance that they’ll need to make outbound connections from the command line?
  • With sendmail, we’ve already discussed whether it allows Incoming SMTP connections on port 25.  What if it needs to deliver a message to a different system?  It would have to make an Outgoing SMTP connection to push the message along.
  • If this system is being backed up, how does that happen?  Is there a program on here that makes an outbound connection each day?  Because we analyzed all the listening ports above, it doesn’t appear there’s a daemon listening for incoming connections.

Even though we have to put some work into reading the netstat output, the advantage of using it is that we get a good picture of what needs to be protected with Incoming firewall rules, and we get some hints of what needs to go in the Outgoing rules too.  We’ve covered TCP ports, and that leaves one more major topic – UDP ports – for the next blog.

Related Posts