A Minimalist Setup to Encrypt Traffic Between Linux Servers Using IPSec-Tools/Racoon

By | April 6, 2014

Let's have two  servers running  Ubuntu Linux, one we  call the red machine,  the other the blue machine, and we want the traffic between both encrypted using IPSec.

To avoid mistakes and to retain a clear view, first we add on all machines (including our workstation) entries for the servers to /etc/hosts. For example, if the red machine has the IP address and the blue machine, /etc/hosts should contain besides entries for localhost and other things the following lines: red.cauwersin.com red blue.cauwersin.com blue

If we start some terminal windows, give them red respectively blue background color, we get a pretty nice view on our machines.


We can test our setup by pinging the red machine from the blue. It should look like this:

root@blue:/etc/racoon# ping red
PING red.cauwersin.com ( 56(84) bytes of data.
64 bytes from red.cauwersin.com ( icmp_req=1 ttl=63 time=0.484 ms
64 bytes from red.cauwersin.com ( icmp_req=2 ttl=63 time=0.472 ms

So far our setup. Let's go to IPSec.

First  we install on both machines (the red and the blue) Racoon. This also brings IPSec-Tools to them. When we get asked during installation for "configuration mode" we choose "direct".

# apt-get install racoon

Now we go to the red machine and modify /etc/ipsec-tools.conf.

#!/usr/sbin/setkey -f
spdadd red blue any -P out ipsec
spdadd blue red any -P in ipsec

Thus we have added security policies which require all outgoing traffic from red to blue and incoming from blue to red to be encrypted. To get these policies into effect, we have to push them into the kernel using setkey.

root@red:/# service setkey restart

Now the blue machine shouldn't be able to ping the red machine anymore, because the red machine only accepts encrypted traffic from the blue machine and would also send only encrypted traffic to the blue machine, but the blue machine doesn't encrypt anything.
To setup encryption on the blue machine we modify there /etc/ipsec-tools as we did on the red machine, except that we flip in and out.

#!/usr/sbin/setkey -f
spdadd red blue any -P in ipsec
spdadd blue red any -P out ipsec

Pushing the IPSec rules into the kernel, lets the blue machine also encrypt traffic.

root@red:/# service setkey restart

Security policies on both machines are now configured.

On both machines we modify  /etc/racoon/racoon.conf as follows, giving the racoon daemon knowledge, how to setup keys:

log notify;
path pre_shared_key "/etc/racoon/psk.txt";
path certificate "/etc/racoon/certs";

remote anonymous {
        exchange_mode main,aggressive;
        proposal {
                encryption_algorithm aes_256;
                hash_algorithm sha256;
                authentication_method pre_shared_key;
                dh_group modp1024;
        generate_policy off;

sainfo anonymous{
        pfs_group 2;
        encryption_algorithm aes_256;
        authentication_algorithm hmac_sha256;
        compression_algorithm deflate;

We are almost done. We just have  to think off a pre-shared key and put it into /etc/racoon/psk.txt on both machines, associated with the IP address of the respective remote machine.

In our case we put in /etc/racoon/psk.txt on the blue machine: Very_Secret_Key

And on the red machine: Very_Secret_Key

Finally we restart the racoon service on both machines:

# service racoon restart

That's it.

Traffic between blue and red is now encrypted.


4 thoughts on “A Minimalist Setup to Encrypt Traffic Between Linux Servers Using IPSec-Tools/Racoon

  1. Red

    Hi - Great walk through. Thanks for taking the time to share it.

    Could one add multiple lines to the psk file (assuming the IPSec Tools were in place an d configured on additional hosts) if we wanted to expand the encryption to apply to communication for two or mote servers?

    e.g. adding a "green" machine and forcing all three to then communicate using encryption.

    1. 7u83 Post author

      You can have multiple lines in the psk.txt file. So adding a "green" machine would be no problem.

  2. Nayan

    Hi, Thanks for this basic tutorial, However I was not able to ping the other machine.
    Apart from the above configurations, should I also configure the Linux OS, kernel with certain modules.


    1. 7u83 Post author

      As far as I know, there are no other modules required.
      I would recommend to re-check the following thing:
      1. Can both machines ping each other with Racoon and IPSec disabled?
      2. Are there any typos in your PSK on both machines or a mistyped IP address in racoon.conf or somewhere else?
      In case, you want to ping some IPs within an internal sub-net on the machines, you can't do that with this setup. Therefore you have to establish a tunnel between the machines. This setup only encrypts traffic between the machines which is send to and received from their external IPs.


Leave a Reply

Your email address will not be published.