Linux Trick 5: SSH back door

Many times I'll be at a site where I need remote support from someone who is blocked on the outside by a company firewall. Few people realize that if you can get out to the world through a firewall, then it is relatively easy to open a hole so that the world can come into you.


In its crudest form, this is called "poking a hole in the firewall." I'll call it an SSH back door. To use it, you'll need a machine on the Internet that you can use as an intermediary.
In our example, we'll call our machine blackbox.example.com. The machine behind the company firewall is called ginger. Finally, the machine that technical support is on will be called tech. Figure 4 explains how this is set up.

Figure 4. Poking a hole in the firewall

 
Here's how to proceed:

1. Check that what you're doing is allowed, but make sure you ask the right people. Most people will cringe that you're opening the firewall, but what they don't understand is that it is completely encrypted. Furthermore, someone would need to hack your outside machine before getting into your company. Instead, you may belong to the school of "ask-for-forgiveness-instead-of-permission." Either way, use your judgment and don't blame me if this doesn't go your way.
   
   
2. SSH from ginger to blackbox.example.com with the -R flag. I'll assume that you're the root user on ginger and that tech will need the root user ID to help you with the system. With the -R flag, you'll forward instructions of port 2222 on blackbox to port 22 on ginger. This is how you set up an SSH tunnel. Note that only SSH traffic can come into ginger: You're not putting ginger out on the Internet naked. You can do this with the following syntax:
   
   ~# ssh -R 2222:localhost:22 thedude@blackbox.example.com
 
   Once you are into blackbox, you just need to stay logged in. I usually enter a command like:
   
   thedude@blackbox:~$ while [ 1 ]; do date; sleep 300; done
 
   to keep the machine busy. And minimize the window.
   
   
3. Now instruct your friends at tech to SSH as thedude into blackbox without using any special SSH flags. You'll have to give them your password:  
root@tech:~# ssh thedude@blackbox.example.com
   
   
   
4. Once tech is on the blackbox, they can SSH to ginger using the following command:  
thedude@blackbox:~$: ssh -p 2222 root@localhost
 
   
5. Tech will then be prompted for a password. They should enter the root password of ginger.
   
   
6. Now you and support from tech can work together and solve the problem. You may even want to use screen together! (See Trick 4.)

Linux Trick 4: Getting back the root password

You forgot your root password. Nice work. Now you'll just have to reinstall the entire machine. Sadly enough, I've seen more than a few people do this. But it's surprisingly easy to get on the machine and change the password. This doesn't work in all cases (like if you made a GRUB password and forgot that too), but here's how you do it in a normal case with a Cent OS Linux example.


First reboot the system. When it reboots you'll come to the GRUB screen as shown in Figure 1. Move the arrow key so that you stay on this screen instead of proceeding all the way to a normal boot.

Figure 1. GRUB screen after reboot

Next, select the kernel that will boot with the arrow keys, and type E to edit the kernel line. You'll then see something like Figure 2:

Figure 2. Ready to edit the kernel line

Use the arrow key again to highlight the line that begins with kernel, and press E to edit the kernel parameters. When you get to the screen shown in Figure 3, simply append the number 1 to the arguments as shown in Figure 3:

Figure 3. Append the argument with the number 1

Then press Enter, B, and the kernel will boot up to single-user mode. Once here you can run the passwd command, changing password for user root:

sh-3.00# passwd
New UNIX password:
Retype new UNIX password:
passwd: all authentication tokens updated successfully 
 
Now you can reboot, and the machine will boot up with your new password.

Linux Trick 3: Collaboration with screen

Now, let's think about the following situation:

David, the high-maintenance user from product engineering, calls: "I need you to help me understand why I can't compile supercode.c on these new machines you deployed."
"Fine," you say. "What machine are you on?"
David responds: " Porsche." OK, you say. You exercise your godlike root powers and on another machine become David:

# su - david
 
Then you go over to posh:

# ssh porsche
 
Once you are there, you run:

# screen -S foo
 
Then you holler at David:

"Hey David, run the following command on your terminal: # screen -x foo."

This will cause your and David's sessions to be joined together in the holy Linux shell. You can type or he can type, but you'll both see what the other is doing. This saves you from walking to the other floor and lets you both have equal control. The benefit is that David can watch your troubleshooting skills and see exactly how you solve problems.


At last you both see what the problem is: David's compile script hard-coded an old directory that does not exist on this new server. You mount it, recompile, solve the problem, and David goes back to work. You then go back to whatever lazy activity you were doing before.


The one caveat to this trick is that you both need to be logged in as the same user. Other cool things you can do with the screen command include having multiple windows and split screens. Read the man pages for more on that.

But I'll give you one last tip while you're in your screen session. To detach from it and leave it open, type: Ctrl-A D. (I mean, hold down the Ctrl key and strike the A key. Then push the D key.)
You can then reattach by running the screen -x foo command again.

Linux Trick 2: Getting your screen back when it's hosed

Try this:

# cat /bin/cat
 
Behold! Your terminal looks like garbage. Everything you type looks like you're looking into the Matrix. What do you do?
You type reset. But wait you say, typing reset is too close to typing reboot or shutdown. Your palms start to sweat—especially if you are doing this on a production machine.
Rest assured: You can do it with the confidence that no machine will be rebooted. Go ahead, do it:

# reset
 
Now your screen is back to normal. This is much better than closing the window and then logging in again, especially if you just went through five machines to SSH to this machine.

Linux Trick 1: Unmounting the unresponsive DVD drive in Linux

The newbie states that when he pushes the Eject button on the DVD drive of a server running a certain Redmond-based operating system, it will eject immediately. He then complains that, in most enterprise Linux servers, if a process is running in that directory, then the ejection won't happen. For too long as a Linux administrator, I would reboot the machine and get my disk on the bounce if I couldn't figure out what was running and why it wouldn't release the DVD drive. But this is ineffective.
Here's how you find the process that holds your DVD drive and eject it to your heart's content: First, simulate it. Stick a disk in your DVD drive, open up a terminal, and mount the DVD drive:

# mount /media/cdrom
# cd /media/cdrom
# while [ 1 ]; do echo "All your drives are belong to us!"; sleep 30; done
 
Now open up a second terminal and try to eject the DVD drive:

# eject
 
You'll get a message like:

umount: /media/cdrom: device is busy

Before you free it, let's find out who is using it.


# fuser /media/cdrom
 
You see the process was running and, indeed, it is our fault we can not eject the disk.
Now, if you are root, you can exercise your godlike powers and kill processes:


# fuser -k /media/cdrom
 
Boom! Just like that, freedom. Now solemnly unmount the drive:


# eject
 
fuser is good.

Openswan-2.6.26: IPSec for Linux

Hello everyone!!!  It’s been a while since the last post but I’m back again ;). I’ve been playing with some old stuff (new for me though) that I will share with all of you today, given how useful this can be for someone like me, looking for how to implement IPSec on a linux box.

Some days ago, my boss ask me to find the way to connect, using IPSec, one of our brand new branch offices, to an specific part on our network protected by an iptables box. He also ask , if possible, to use a web interface to manage both, the iptables and IPSec implementation, since we are running short on personnel and a web interface can make adding, removing and editing connections very easy for a newbie. On the branch side they were using a Linksys 4-Port VPN Router RV042, which only has basic IPSec capabilities built in into the Router(I hope Linksys include openvpn capabilities soon) so there was no openvpn choice in this case.

At the beginning, all I knew was that there were a couple of choices for this that could help me make this happen, but I never needed to use any of them before, so I dive into some research before starting with the configuration. Long time ago, the IPSec support in Linux was provided by freeswan. FreeS/WAN, for Free Secure Wide-Area Networking, was a free software project, which implemented a reference version of the IPsec network security layer for Linux and other Unix-like operating systems. The main project goal of ubiquitous opportunistic encryption of Internet traffic was not realized, although it did contribute to general Internet encryption. The final FreeS/WAN version 2.06 was released on 22 April 2004, while the earlier version 2.04 was forked to form two projects, Openswan  and strongSwan.

After a little more research, I find out that the focus of the strongSwan project is on strong authentication mechanisms using X.509 public key certificates and optional secure storage of private keys on smartcards through a standardized PKCS#11 interface. It supports certificate revocation lists and the Online Certificate Status Protocol (OCSP). A unique feature is the use of X.509 attribute certificates to implement advanced access control schemes based on group memberships. So given the simplicity of the task I was assigned, this option was way too complex, and it was not supported by the IPSec implementation built into the Linksys firewall that the branch office was using. Then openswan turn out to be the way to go, given that it is a complete IPsec implementation for Linux 2.0, 2.2, 2.4 and 2.6 kernels and that it is currently being actively developed and supported by the openswan project.

Installing openswan-2.6.26

The installation of openswan is very straight forward for almost any Linux distribution, since it is included in the distro media and it is present in every linux repositories. Therefore rpm and deb packages are available for downloading and installation from all over the Internet. Yum and apt-get can also be used for the installation under any major Linux distro. In my case, we are using CentOS 5.5 for the iptables box and this will be distro I will use to give the details of the setup process.

This box was deployed as a CentOS minimal install (although it is 800 MB). The way to do this during the installation process is by unselecting all the packages selected by default by the installer. I couldn’t find a way to make an installation of CentOS 5.5 smaller than this. Before proceeding with the installation of openswan under CentOS, I needed to add a package that will allow the web interface selected to manage the openswan IPSec implementation.

yum install nss_db-2.2-35.3 openswan

At the moment openswan is being installed, a new dialog will appear asking if you want to generate a new RSA key signature, which I avoided by selecting cancel since I would do this later from the command line.

Setting Up openswan

After openswan is installed on the system, the service must be started and then the first thing you should do is verifying the installation with the following commands:

service ipsec start

ipsec verify

Most probably you will get some “FAILED” warnings after issuing the previous commands. Something like this:

[root@fw-linux ~]# ipsec verify
Checking your system to see if IPsec got installed and started correctly:
Version check and ipsec on-path                                                                                                       [OK]
Linux Openswan U2.6.21/K2.6.18-164.el5 (netkey)
Checking for IPsec support in kernel                                                                                                [OK]
NETKEY detected, testing for disabled ICMP send_redirects                                                  [FAILED]

  Please disable /proc/sys/net/ipv4/conf/*/send_redirects
  or NETKEY will cause the sending of bogus ICMP redirects!

NETKEY detected, testing for disabled ICMP accept_redirects                                               [FAILED]

  Please disable /proc/sys/net/ipv4/conf/*/accept_redirects
  or NETKEY will accept bogus ICMP redirects!

Checking for RSA private key (/etc/ipsec.secrets)                                                                             [OK]
Checking that pluto is running                                                                                                          [OK]
Two or more interfaces found, checking IP forwarding                                                           [FAILED]
Checking for 'ip' command                                                                                                                [OK]
Checking for 'iptables' command                                                                                                       [OK]

Opportunistic Encryption DNS checks:
   Looking for TXT in forward dns zone: localhost.domain.com                                             [MISSING]
   Does the machine have at least one non-private address?                                                             [OK]
  

This “FAILED” warnings, should be solved before proceeding with the openswan setup. For the warnings related to send_redirects,  accept_redirects and IP forwarding edit the /etc/sysctl.conf file and the following lines should be added to the bottom of the file:

# IPSec Verify Compliant
net.ipv4.conf.default.send_redirects=0
net.ipv4.conf.all.send_redirects=0
net.ipv4.conf.eth0.send_redirects=0
net.ipv4.conf.eth1.send_redirects=0
net.ipv4.conf.lo.send_redirects=0
net.ipv4.conf.default.accept_redirects=0
net.ipv4.conf.all.accept_redirects=0
net.ipv4.conf.eth0.accept_redirects=0
net.ipv4.conf.eth1.accept_redirects=0
net.ipv4.conf.lo.accept_redirects=0

Usually using openswan, involves some kind of routing as well, that’s why net.ipv4.ip_forward default value 0 should be changed to 1 under the same /etc/sysctl.conf file.

net.ipv4.ip_forward = 1

After this has been done, ipsec verify should be run one more time to be sure it is not complaining any more.

[root@fw-linux ~]# ipsec verify
Checking your system to see if IPsec got installed and started correctly:
Version check and ipsec on-path                                                                                                       [OK]
Linux Openswan U2.6.21/K2.6.18-164.el5 (netkey)
Checking for IPsec support in kernel                                                                                                [OK]
NETKEY detected, testing for disabled ICMP send_redirects                                                          [OK]
NETKEY detected, testing for disabled ICMP accept_redirects                                                       [OK]
Checking for RSA private key (/etc/ipsec.secrets)                                                                             [OK]
Checking that pluto is running                                                                                                          [OK]
Two or more interfaces found, checking IP forwarding                                                                    [OK]
Checking NAT and MASQUERADEing                                                                                             [OK]
Checking for 'ip' command                                                                                                                [OK]
Checking for 'iptables' command                                                                                                       [OK]

Opportunistic Encryption DNS checks:
Looking for TXT in forward dns zone: localhost.domain.com                                                [MISSING]
   Does the machine have at least one non-private address?                                                             [OK]

The Oportunistic Encryption warnings can be ignored, since this is not critical for the system and I don’t need to make use of it in this setup. If wanted, opportunistic encryption can be disabled under /etc/ipsec.conf. After checking everything else is “OK” the setup process can continue.

The next step will be generating the new RSA key required for the web interface to work properly, even when we will not be using this method of encryption for our setup. Execute the following command:

ipsec newhostkey --output /etc/ipsec.secrets --bits 2048 --verbose --configdir /etc/pki/nssdb/

The ipsec.secrets file will hold this key as well as the Preshared Key (PSK) that we will be using on our setup. After the previous command finished its execution (it can take a while depending on your hardware) you can check that the /etc/ipsec.secrets contains something similar to this:

: RSA   {
        # RSA 2048 bits   host.example.com   Thu Jun  3 14:34:52 2010
        # for signatures only, UNSAFE FOR ENCRYPTION
        #pubkey=0sAQO4eBafLvzkV0oPfaUYXrAQZ+PpCrTBB8kdcDC4VqEN2fTHe1eo6ac1q28zTiUzskK8tUWrwbG4Zz/Iklnqyh2s5Ci3NeI1TA9mYY8LKul62lrG0nlc2oYxPdMx7VyEMEfYRBdbZph3sZLg8hhXVNcCPtFE0dH1cGJLQt2MzWsxZuZAsA95xaAjNgg9zEbdJ0SyUGFRLUHrAj1eNnfLfOUxegFOU2qObYdCR2sfxk20W1b+a3XofKoWrvf/ojxh8WOn9pPeni3jcbZE2OgKe5yES6jRcCyMIXTtH/v5584PZyUXWJbnoOC/6sdqwQAudu7MZgtsPSYC/GSwGg/jUJWB
        Modulus: 0xb878169f2efce4574a0f7da5185eb01067e3e90ab4c107c91d7030b856a10dd9f4c77b57a8e9a735ab6f334e2533b242bcb545abc1b1b8673fc89259eaca1dace428b735e2354c0f66618f0b2ae97ada5ac6d2795cda86313dd331ed5c843047d844175b669877b192e0f2185754d7023ed144d1d1f570624b42dd8ccd6b3166e640b00f79c5a02336083dcc46dd2744b25061512d41eb023d5e3677cb7ce5317a014e536a8e6d8742476b1fc64db45b56fe6b75e87caa16aef7ffa23c61f163a7f693de9e2de371b644d8e80a7b9c844ba8d1702c8c2174ed1ffbf9e7ce0f6725175896e7a0e0bfeac76ac1002e76eecc660b6c3d2602fc64b01a0fe3509581
        PublicExponent: 0x03
        # everything after this point is CKA_ID in hex format when using NSS
        PrivateExponent: 0x5819408f4506e2ca9552c76f96e971201b425d91
        Prime1: 0x5819408f4506e2ca9552c76f96e971201b425d91
        Prime2: 0x5819408f4506e2ca9552c76f96e971201b425d91
        Exponent1: 0x5819408f4506e2ca9552c76f96e971201b425d91
        Exponent2: 0x5819408f4506e2ca9552c76f96e971201b425d91
        Coefficient: 0x5819408f4506e2ca9552c76f96e971201b425d91
        CKAIDNSS: 0x5819408f4506e2ca9552c76f96e971201b425d91
        }
# do not change the indenting of that "}"

DO NOT CHANGE ANYTHING IN THIS FILE.

The next step would be to edit /etc/ipsec.conf and start including all the specific setting for the tunnel. Before doing this you should know that the ipsec.conf file is divided in three main sections:

• IPSec Version
• Basic or Default Configuration
• Connection Setup

The IPSec Version section contains version 2.0 by default and you should not have to change that.

The 2nd section should look like this:

# basic configuration
config setup
        forwardcontrol=no
        nat_traversal=yes
        nhelpers=0
        oe=off
        protostack=netkey

Keep in mind that the indenting is important to the ipsec daemon, so please keep the indentation in the file if you need to modify any of these fields.

The 3rd section of the file is the one related to the IPSec connection itself. In here you should specify the details of the IPSec connection to match you setup. In my case that section looks like this:

conn Test
        auth=esp
        authby=secret
        auto=start
        compress=no
        esp=3des-md5
        left=1.2.3.4
        leftsubnet=192.168.2.0/24
        pfs=yes
        right=5.6.7.8
        rightsubnet=192.168.1.0/24
        type=tunnel


In this case “Test” is the name I used for the IPSec connection. This can be whatever name you feel like calling you tunnel. As it is seen, this file describes the specifics of the Site-To-Site tunnel between the branch (right) and the main office (left).

The tunnel will use a Previously Shared Secret for the initial negotiation, then it will use Encapsulated Security Payload (esp=3des-md5), Internet Key Exchange(ike=3des-md5), Perfect Forward Secrecy (pfs=yes), no Compression (compress=no), etc. You will find that some of the fields are missing in the file as, for instance, ike=3des-md5. The reason why this happens is because the ipsec daemon was thought to work with the minimal configuration possible, thus if a specific field is defined in the file, the options of that field will be restrained to those options only, while if a field is not defined, the IPSec daemon will try every single possibility for that specific field. Therefore the only fields that are mandatory on this file are: left, leftsubnet, right, rightsubnet, auto and authby.

I have found that sometimes it is best to define explicitly every field (the tunnel doesn’t get established otherwise), since this will assure that the options used are the ones you want, giving you more control over the IPSec connection.

Openswan integration with Webmin

As my boss also requested a web interface to manage the IPSec setup on this box, I decided to use webmin, because it has been around for a while now and also because its modular architecture will allow for any kind of plugging (even one developed by myself if needed). Luckily there was one module, originally written for freeSwan and named IPSec VPN Configuration, that is fully compatible with openswan.

The webmin installation process is pretty straight forward as well, given that webmin contains pretty much everything it needs to work properly. So, basically the steps are: download the webmin application and the install it.

To download webmin for Centos execute:

wget http://prdownloads.sourceforge.net/webadmin/webmin-1.510-1.noarch.rpm

After the download finishes, execute the following command to install webmin:

rpm -ivh webmin-1.510-1.noarch.rpm

Now open your browser and go to: http://yourserveripaddress:1000 and log in using the root credentials. The IPSec VPN Configuration module will show under the “Network” section on the left side of your browser. In my case, I first had to modify the path for the IPSec VPN Configuration module on webmin to match the path where the ipsec binary is on my system, in this case /usr/local/sbin. So, if this is the case as well, then go to the search box on webmin and type: “ipsec”. The IPSec VPN Configuration module for webmin should appear under the search results and from there you will be able to modify the module configuration and change the path to the ipsec binary.
After this, the IPSec VPN Module should be shown under the Networking Section like this:

Using this interface, you should be able to define your tunnel specific details by adding a new IPSec VPN Connection, and after that you should also be able to define the PSK under Secret Keys that is going to be used o the tunnel. Keep in mind that after you have entered all the information needed for the tunnel, you need to click on the “Apply configuration” button first and then, click on the “Start Connection” button of this interface to bring the tunnel up.

Troubleshooting openswan

Almost every tunnel I have set up in my life needed some kind of troubleshooting, since, most of the times, those tunnels don’t come up as they were supposed to once you set it up, and this is where the troubleshooting process is important. There are three basic ways to troubleshoot a tunnel. The first one is, of course, using tcpdump to check for the ISAKMP negotiations and later for the ESP packet. The second choice will be checking the ipsec daemon log, which typically is under /var/log/secure by default, and look for “IPsec SA established”. If your logs contains this string, then you tunnel is up and running. This is my favorite choice for troubleshooting the tunnel since you can get a lot of information about what’s going on with the tunnel negotiation between endpoints. The third choice can be issuing the ipsec setup --status command on the CLI.

Openswan and Iptables

There are a number of things that I will like to highlight with regard to iptables and openswan. First, keep in mind that for the tunnel to be established, you will need a couple of rules on your iptables that will allow the tunnel endpoints to communicate properly with each other. The IPSec Phase 1 uses udp port 500 on both endpoints to initiate the communication and for the Internet Key Exchange. After this phase takes place successfully, the endpoints need to communicate with each other using the ESP protocol, to complete the phase 2 of the IPSec protocol. To allow the IPSec connections between the endpoint, you should add the following rules on the iptables:

iptables -A INPUT -s 5.6.7.8 -d 1.2.3.4 -p udp -m udp --sport 500 --dport 500 -j ACCEPT
iptables -A INPUT -s 5.6.7.8 -d 1.2.3.4 -p esp -j ACCEPT
iptables -A OUTPUT -s 5.6.7.8 -d1.2.3.4 -p udp -m udp --sport 500 --dport 500 -j ACCEPT
iptables -A OUTPUT -s 5.6.7.8 -d 1.2.3.4 -p esp -j ACCEPT

The modern implementations of openswan on 2.6 kernels make use of the NETKEY module built into the kernel itself. Basically what this does is that once the packet has been decrypted by openswan, openswan give back the decrypted packet to the kernel so it can route it to the proper destination, by putting the packet on the same interface it arrived, but this time without encryption. Essentially this allow to filter the packets using iptables as well, but of course, using the leftsubnet and rightsubnet source an destination ip address respectively, instead of the ip addresses of the endpoints. In this case we wan to only allow the PCs behind the right side (remote side) of the tunnel to connect to a specific PC behind the left side (local side) of the tunnel. For that purpose, the following iptables rules where used:

iptables -A FORWARD -s 192.168.1.0/255.255.255.0 -d 192.168.2.2/255.255.255.255 -j ACCEPT
iptables -A FORWARD -s 192.168.2.2/255.255.255.255 -d 192.168.1.0/255.255.255.0 -j ACCEPT

All the changes to iptables can be done using the Webmin interface as well, which adds value to the implementation.

Conclusions

After some days of research and testing, we came up with a fairly small system that can be used as a mean to protect and interconnect a specific portion of our internal network. This will allow us to interoperate with branches or client, using fully standardized and compatibles protocols such as IPSec, which is normally supported on most of the commercial firewalls in the market today, at a very cheap price.

Webmin once again end up being the web management interface of choice, when this is needed because of its modularity, its friendly interface and its portability to almost every Linux distribution.

I hope you guys find this helpful or at least informative. Please, feel free to leave any comments, thoughts, suggestion or contributions below.

Thanks !