Wednesday, June 9, 2010

Openswan-2.6.26: IPSec for Linux

Hello everyone!!!  It’s been a while since the last post but I’m back again ;). I’ve been playing with some old stuff (new for me though) that I will share with all of you today, given how useful this can be for someone like me, looking for how to implement IPSec on a linux box.

Some days ago, my boss ask me to find the way to connect, using IPSec, one of our brand new branch offices, to an specific part on our network protected by an iptables box. He also ask , if possible, to use a web interface to manage both, the iptables and IPSec implementation, since we are running short on personnel and a web interface can make adding, removing and editing connections very easy for a newbie. On the branch side they were using a Linksys 4-Port VPN Router RV042, which only has basic IPSec capabilities built in into the Router(I hope Linksys include openvpn capabilities soon) so there was no openvpn choice in this case.

At the beginning, all I knew was that there were a couple of choices for this that could help me make this happen, but I never needed to use any of them before, so I dive into some research before starting with the configuration. Long time ago, the IPSec support in Linux was provided by freeswan. FreeS/WAN, for Free Secure Wide-Area Networking, was a free software project, which implemented a reference version of the IPsec network security layer for Linux and other Unix-like operating systems. The main project goal of ubiquitous opportunistic encryption of Internet traffic was not realized, although it did contribute to general Internet encryption. The final FreeS/WAN version 2.06 was released on 22 April 2004, while the earlier version 2.04 was forked to form two projects, Openswan  and strongSwan.

After a little more research, I find out that the focus of the strongSwan project is on strong authentication mechanisms using X.509 public key certificates and optional secure storage of private keys on smartcards through a standardized PKCS#11 interface. It supports certificate revocation lists and the Online Certificate Status Protocol (OCSP). A unique feature is the use of X.509 attribute certificates to implement advanced access control schemes based on group memberships. So given the simplicity of the task I was assigned, this option was way too complex, and it was not supported by the IPSec implementation built into the Linksys firewall that the branch office was using. Then openswan turn out to be the way to go, given that it is a complete IPsec implementation for Linux 2.0, 2.2, 2.4 and 2.6 kernels and that it is currently being actively developed and supported by the openswan project.

Installing openswan-2.6.26

The installation of openswan is very straight forward for almost any Linux distribution, since it is included in the distro media and it is present in every linux repositories. Therefore rpm and deb packages are available for downloading and installation from all over the Internet. Yum and apt-get can also be used for the installation under any major Linux distro. In my case, we are using CentOS 5.5 for the iptables box and this will be distro I will use to give the details of the setup process.

This box was deployed as a CentOS minimal install (although it is 800 MB). The way to do this during the installation process is by unselecting all the packages selected by default by the installer. I couldn’t find a way to make an installation of CentOS 5.5 smaller than this. Before proceeding with the installation of openswan under CentOS, I needed to add a package that will allow the web interface selected to manage the openswan IPSec implementation.

yum install nss_db-2.2-35.3 openswan

At the moment openswan is being installed, a new dialog will appear asking if you want to generate a new RSA key signature, which I avoided by selecting cancel since I would do this later from the command line.

Setting Up openswan

After openswan is installed on the system, the service must be started and then the first thing you should do is verifying the installation with the following commands:

service ipsec start

ipsec verify

Most probably you will get some “FAILED” warnings after issuing the previous commands. Something like this:

[root@fw-linux ~]# ipsec verify
Checking your system to see if IPsec got installed and started correctly:
Version check and ipsec on-path                                                                                                       [OK]
Linux Openswan U2.6.21/K2.6.18-164.el5 (netkey)
Checking for IPsec support in kernel                                                                                                [OK]
NETKEY detected, testing for disabled ICMP send_redirects                                                  [FAILED]

  Please disable /proc/sys/net/ipv4/conf/*/send_redirects
  or NETKEY will cause the sending of bogus ICMP redirects!

NETKEY detected, testing for disabled ICMP accept_redirects                                               [FAILED]

  Please disable /proc/sys/net/ipv4/conf/*/accept_redirects
  or NETKEY will accept bogus ICMP redirects!

Checking for RSA private key (/etc/ipsec.secrets)                                                                             [OK]
Checking that pluto is running                                                                                                          [OK]
Two or more interfaces found, checking IP forwarding                                                           [FAILED]
Checking for 'ip' command                                                                                                                [OK]
Checking for 'iptables' command                                                                                                       [OK]

Opportunistic Encryption DNS checks:
   Looking for TXT in forward dns zone:                                             [MISSING]
   Does the machine have at least one non-private address?                                                             [OK]


This “FAILED” warnings, should be solved before proceeding with the openswan setup. For the warnings related to send_redirects,  accept_redirects and IP forwarding edit the /etc/sysctl.conf file and the following lines should be added to the bottom of the file:

# IPSec Verify Compliant

Usually using openswan, involves some kind of routing as well, that’s why net.ipv4.ip_forward default value 0 should be changed to 1 under the same /etc/sysctl.conf file.

net.ipv4.ip_forward = 1

After this has been done, ipsec verify should be run one more time to be sure it is not complaining any more.

[root@fw-linux ~]# ipsec verify
Checking your system to see if IPsec got installed and started correctly:
Version check and ipsec on-path                                                                                                       [OK]
Linux Openswan U2.6.21/K2.6.18-164.el5 (netkey)
Checking for IPsec support in kernel                                                                                                [OK]
NETKEY detected, testing for disabled ICMP send_redirects                                                          [OK]
NETKEY detected, testing for disabled ICMP accept_redirects                                                       [OK]
Checking for RSA private key (/etc/ipsec.secrets)                                                                             [OK]
Checking that pluto is running                                                                                                          [OK]
Two or more interfaces found, checking IP forwarding                                                                    [OK]
Checking NAT and MASQUERADEing                                                                                             [OK]
Checking for 'ip' command                                                                                                                [OK]
Checking for 'iptables' command                                                                                                       [OK]

Opportunistic Encryption DNS checks:
Looking for TXT in forward dns zone:                                                [MISSING]
   Does the machine have at least one non-private address?                                                             [OK]

The Oportunistic Encryption warnings can be ignored, since this is not critical for the system and I don’t need to make use of it in this setup. If wanted, opportunistic encryption can be disabled under /etc/ipsec.conf. After checking everything else is “OK” the setup process can continue.

The next step will be generating the new RSA key required for the web interface to work properly, even when we will not be using this method of encryption for our setup. Execute the following command:

ipsec newhostkey --output /etc/ipsec.secrets --bits 2048 --verbose --configdir /etc/pki/nssdb/

The ipsec.secrets file will hold this key as well as the Preshared Key (PSK) that we will be using on our setup. After the previous command finished its execution (it can take a while depending on your hardware) you can check that the /etc/ipsec.secrets contains something similar to this:

: RSA   {
        # RSA 2048 bits   Thu Jun  3 14:34:52 2010
        # for signatures only, UNSAFE FOR ENCRYPTION
        Modulus: 0xb878169f2efce4574a0f7da5185eb01067e3e90ab4c107c91d7030b856a10dd9f4c77b57a8e9a735ab6f334e2533b242bcb545abc1b1b8673fc89259eaca1dace428b735e2354c0f66618f0b2ae97ada5ac6d2795cda86313dd331ed5c843047d844175b669877b192e0f2185754d7023ed144d1d1f570624b42dd8ccd6b3166e640b00f79c5a02336083dcc46dd2744b25061512d41eb023d5e3677cb7ce5317a014e536a8e6d8742476b1fc64db45b56fe6b75e87caa16aef7ffa23c61f163a7f693de9e2de371b644d8e80a7b9c844ba8d1702c8c2174ed1ffbf9e7ce0f6725175896e7a0e0bfeac76ac1002e76eecc660b6c3d2602fc64b01a0fe3509581
        PublicExponent: 0x03
        # everything after this point is CKA_ID in hex format when using NSS
        PrivateExponent: 0x5819408f4506e2ca9552c76f96e971201b425d91
        Prime1: 0x5819408f4506e2ca9552c76f96e971201b425d91
        Prime2: 0x5819408f4506e2ca9552c76f96e971201b425d91
        Exponent1: 0x5819408f4506e2ca9552c76f96e971201b425d91
        Exponent2: 0x5819408f4506e2ca9552c76f96e971201b425d91
        Coefficient: 0x5819408f4506e2ca9552c76f96e971201b425d91
        CKAIDNSS: 0x5819408f4506e2ca9552c76f96e971201b425d91
# do not change the indenting of that "}"


The next step would be to edit /etc/ipsec.conf and start including all the specific setting for the tunnel. Before doing this you should know that the ipsec.conf file is divided in three main sections:
  • IPSec Version
  • Basic or Default Configuration
  • Connection Setup

The IPSec Version section contains version 2.0 by default and you should not have to change that.

The 2nd section should look like this:

# basic configuration
config setup

Keep in mind that the indenting is important to the ipsec daemon, so please keep the indentation in the file if you need to modify any of these fields.

The 3rd section of the file is the one related to the IPSec connection itself. In here you should specify the details of the IPSec connection to match you setup. In my case that section looks like this:

conn Test

In this case “Test” is the name I used for the IPSec connection. This can be whatever name you feel like calling you tunnel. As it is seen, this file describes the specifics of the Site-To-Site tunnel between the branch (right) and the main office (left).

The tunnel will use a Previously Shared Secret for the initial negotiation, then it will use Encapsulated Security Payload (esp=3des-md5), Internet Key Exchange(ike=3des-md5), Perfect Forward Secrecy (pfs=yes), no Compression (compress=no), etc. You will find that some of the fields are missing in the file as, for instance, ike=3des-md5. The reason why this happens is because the ipsec daemon was thought to work with the minimal configuration possible, thus if a specific field is defined in the file, the options of that field will be restrained to those options only, while if a field is not defined, the IPSec daemon will try every single possibility for that specific field. Therefore the only fields that are mandatory on this file are: left, leftsubnet, right, rightsubnet, auto and authby.

I have found that sometimes it is best to define explicitly every field (the tunnel doesn’t get established otherwise), since this will assure that the options used are the ones you want, giving you more control over the IPSec connection.

Openswan integration with Webmin

As my boss also requested a web interface to manage the IPSec setup on this box, I decided to use webmin, because it has been around for a while now and also because its modular architecture will allow for any kind of plugging (even one developed by myself if needed). Luckily there was one module, originally written for freeSwan and named IPSec VPN Configuration, that is fully compatible with openswan.

The webmin installation process is pretty straight forward as well, given that webmin contains pretty much everything it needs to work properly. So, basically the steps are: download the webmin application and the install it.

To download webmin for Centos execute:


After the download finishes, execute the following command to install webmin:

rpm -ivh webmin-1.510-1.noarch.rpm

Now open your browser and go to: http://yourserveripaddress:1000 and log in using the root credentials. The IPSec VPN Configuration module will show under the “Network” section on the left side of your browser. In my case, I first had to modify the path for the IPSec VPN Configuration module on webmin to match the path where the ipsec binary is on my system, in this case /usr/local/sbin. So, if this is the case as well, then go to the search box on webmin and type: “ipsec”. The IPSec VPN Configuration module for webmin should appear under the search results and from there you will be able to modify the module configuration and change the path to the ipsec binary.
After this, the IPSec VPN Module should be shown under the Networking Section like this:

Using this interface, you should be able to define your tunnel specific details by adding a new IPSec VPN Connection, and after that you should also be able to define the PSK under Secret Keys that is going to be used o the tunnel. Keep in mind that after you have entered all the information needed for the tunnel, you need to click on the “Apply configuration” button first and then, click on the “Start Connection” button of this interface to bring the tunnel up.

Troubleshooting openswan

Almost every tunnel I have set up in my life needed some kind of troubleshooting, since, most of the times, those tunnels don’t come up as they were supposed to once you set it up, and this is where the troubleshooting process is important. There are three basic ways to troubleshoot a tunnel. The first one is, of course, using tcpdump to check for the ISAKMP negotiations and later for the ESP packet. The second choice will be checking the ipsec daemon log, which typically is under /var/log/secure by default, and look for “IPsec SA established”. If your logs contains this string, then you tunnel is up and running. This is my favorite choice for troubleshooting the tunnel since you can get a lot of information about what’s going on with the tunnel negotiation between endpoints. The third choice can be issuing the ipsec setup --status command on the CLI.

Openswan and Iptables

There are a number of things that I will like to highlight with regard to iptables and openswan. First, keep in mind that for the tunnel to be established, you will need a couple of rules on your iptables that will allow the tunnel endpoints to communicate properly with each other. The IPSec Phase 1 uses udp port 500 on both endpoints to initiate the communication and for the Internet Key Exchange. After this phase takes place successfully, the endpoints need to communicate with each other using the ESP protocol, to complete the phase 2 of the IPSec protocol. To allow the IPSec connections between the endpoint, you should add the following rules on the iptables:

iptables -A INPUT -s -d -p udp -m udp --sport 500 --dport 500 -j ACCEPT
iptables -A INPUT -s -d -p esp -j ACCEPT
iptables -A OUTPUT -s -d1.2.3.4 -p udp -m udp --sport 500 --dport 500 -j ACCEPT
iptables -A OUTPUT -s -d -p esp -j ACCEPT

The modern implementations of openswan on 2.6 kernels make use of the NETKEY module built into the kernel itself. Basically what this does is that once the packet has been decrypted by openswan, openswan give back the decrypted packet to the kernel so it can route it to the proper destination, by putting the packet on the same interface it arrived, but this time without encryption. Essentially this allow to filter the packets using iptables as well, but of course, using the leftsubnet and rightsubnet source an destination ip address respectively, instead of the ip addresses of the endpoints. In this case we wan to only allow the PCs behind the right side (remote side) of the tunnel to connect to a specific PC behind the left side (local side) of the tunnel. For that purpose, the following iptables rules where used:

iptables -A FORWARD -s -d -j ACCEPT
iptables -A FORWARD -s -d -j ACCEPT

All the changes to iptables can be done using the Webmin interface as well, which adds value to the implementation.


After some days of research and testing, we came up with a fairly small system that can be used as a mean to protect and interconnect a specific portion of our internal network. This will allow us to interoperate with branches or client, using fully standardized and compatibles protocols such as IPSec, which is normally supported on most of the commercial firewalls in the market today, at a very cheap price.

Webmin once again end up being the web management interface of choice, when this is needed because of its modularity, its friendly interface and its portability to almost every Linux distribution.

I hope you guys find this helpful or at least informative. Please, feel free to leave any comments, thoughts, suggestion or contributions below.

Thanks !


  1. Thanks a lot.Very useful post.
    Good advices and schemas for ipsec VPN on Linux.
    Nice VPN client.

  2. This comment has been removed by the author.

  3. This comment has been removed by the author.

  4. Great when using my Denver VPS or Los Angeles VPS to watch US movies and TV series!