System administrators often have to copy data around. Copying and converting ordinary data is easily accomplished with the Linux command called cp. However, if the data is not ordinary, cp is not powerful enough. The needed power can be found in the dd command, and here are some ways to put that power to good use.
The dd command handles convert-and-copy tasks. Obviously, "cc" would have been a better name, but there already was a command with that name when dd was invented. It doesn't matter since it's a cool command.
Cloning a hard drive with dd
The dd command doesn't just copy files; it copies blocks, too. As a simple example, I'll show you how to clone your entire hard drive.
Assuming that /dev/sda is the drive that you want to clone, and /dev/sdb is an empty drive that can be used as the target, using the dd command rather easy:
dd if=/dev/sda of=/dev/sdb
In that example, dd is used with only two parameters: if, which is used to specify an input file, and of, which is used to specify the output file. In this case, both input and output files are device files. Wait until the command is finished, and you will end up with an exact copy of the original hard drive.
In the latter example, the contents of a device were copied to another device. A slight variation to that is the way that dd is used to clone a DVD or CD-ROM and write it to an ISO file. If your optical drive can be accessed via /dev/cdrom, then you can clone the optical disk using:
dd if=/dev/cdrom of=/tmp/cdrom.iso
You can also mount that ISO-file using:
mount -o loop /tmp/cdrom.iso /mnt
Next, you can access the files in the ISO-file from the directory where the ISO is mounted.
Creating a backup of the Master Boot Record
So far, we have used dd to perform tasks that can be done with other utilities as well. Now, we can go beyond that. In the following example, let's create a backup of the Master Boot Record (MBR).
Copy the first 512 bytes of your hard drive, which contains the MBR, to a file. For instance, you could do this with the command
dd if=/dev/sda of=/boot/mbr_backup bs=512 count=1
In this scenario, two new parameters are used:
First, there's the parameter bs=512, which specifies that a blocksize of 512 bytes should be used.
Next, the parameter 1 is used to indicate that only one such block has to be copied. Without that parameter, you would copy your entire hard drive.
The backup copy of your MBR may be useful if, some day, you can't boot your server anymore because of a problem in the MBR. In case that happens, just boot from a rescue disk and use the command:
dd if=/boot/mbr_backup of=/dev/sda bs=446 count=1
In this restore command, only 446 bytes are written back. This is because you may have changed the partition table since you've created the backup. By writing back only the first 446 bytes of your backup file, you don't overwrite the original partition table, which is between bytes 447 and 511.
Extending swap space
In the last example of how dd can save your life, I'll show you how to extend your swap space by adding a swap file.
Let's say that you're alerted at 3 a.m. by a message saying your server is about to run out of memory entirely, due to an undiscovered memory leak. All you have to do is create an empty file and specify that it should be added to the swap space.
Creating this empty file is an excellent task for dd:
dd if=/dev/zero of=/swapfile bs=1024 count=1000000
This would write a file of one gigabyte, and that can be added to the swap space using:
mkswap /swapfile
and
swapon /swapfile
In this article, you've learned how to do some basic troubleshooting using the dd utility. As you have read, dd is a very versatile utility that goes far beyond the capabilities of ordinary copy tools like cp. Its abilities to work with blocks instead of files are especially valuable.
Tuesday, October 26, 2010
Using grep and sed to find and replace
This article covers two beneficial Linux tools, grep and sed. If searching for or handling text, grep and sed can increase your efficiency with Linux bash scripting and configuration files. We'll learn how to use these commands and get some helpful tips, too.
Using Grep
Need to find a single matching line in a 300 page script or a large configuration file that you've customized? Don't have the time to search line by line to find that single bug that is halting your system when trying to start the service? You can search for a string quickly using grep.
You can search with grep (an acronym for searching globally for lines matching the regular expression, and print them) many of the files on a Linux machine. When you give it a list of files to read, it will search for lines of text that match the expressions you're looking for and, if it find any, it will output the pieces you want. Just think of as a grip: using grep is a great way to grab a hold of specific lines or files. This command line utility was originally designed for Unix systems.
For example, let's say we have a list of numbers in a file, numlist.txt, and we want to see if the number 8723 is in the list without searching for hours. Using grep can easily find the string in the following command line:
grep –i 8723 numlist.txt
If we get an output of 8723, then it is in the list. If we do not get an output, it is not there. The tool will print the whole line where the text string is found. If the list contained 8723 297r, we would get:
[root@tower root]# grep 8723 numlist.txt 8723 297r
[root@tower root]#
Grep is case sensitive by default. To disable this, use the –i flag.
Here's something useful in a daily RHEL system checkup. Say you need to find out which version of sendmail rpm you installed. To list which rpm packages are installed on the system, you enter the following:
rpm –qa
If you have a system like mine, you have just too many packages to sort through. It might take forever to go through them line by line to find the package you're looking for. Wouldn't it be easier to find a line with a text string matching the word "sendmail?" Thankfully, we can use grep like this:
rpm –qa | grep sendmail
Which should give you:
[root@tower root]# rpm -qa | grep sendmail sendmail-8.13.1-3.RHEL4.5
[root@tower root]#
Don't know if the string is capitalized or not? Just turn off the case-sensitive default:
rpm –qa | grep –i sendmail
Here's another example of how handy grep is. If we want some information on a process that is running on our server, we would run the command ps, which returns a fairly large list of running processes. To find the certain process that we're looking for, put the grep command after ps. Say I need to make sure my DHCP Server is running. I'll run the following:
ps ax | grep –i dhcp
You should see something like the following:
3058 ? Ss 0:00 /usr/sbin/dhcpd -cf /etc/dhcp/dhcpd.conf -q -pf /var/run/dhcp/dhcpd.pid -user dhcp -group dhcp
All set. Our DHCP server is running thanks to the simple and efficient grep.
Using sed
The command sed (stream editor) is used to take a file, read a line, and change the line as specified by the command. This is another former UNIX utility available today in most Linux distributions.
Let's say we have a file called sedtest.txt, which contains the string "1.0". We want to modify this file, changing the string to "2.0". We can do this by doing the following:
sed -e 's/1.0/2.0/g' sedtest.txt > sedtest2.txt mv –f sedtest2.txt sedtest.txt
In this command line, s stands for substitute and g stands for global. Sedtest.txt is the input file (the one we're changing) and sedtest2.txt is the file we've created from the changes of sedtest.txt. The second command takes the output file and renames it to the input file name (replacing the changes made)
Besides substitution, there are other things sed is useful for. For example, the following deletes empty lines or lines that only contain spaces:
sed -e '/^ *$/d' sedtest1.txt
Notice how this one does not need an output file.
Just like the Replace function in Word and Wordpad, sed is the Linux version of this handy tool.
Using Grep
Need to find a single matching line in a 300 page script or a large configuration file that you've customized? Don't have the time to search line by line to find that single bug that is halting your system when trying to start the service? You can search for a string quickly using grep.
You can search with grep (an acronym for searching globally for lines matching the regular expression, and print them) many of the files on a Linux machine. When you give it a list of files to read, it will search for lines of text that match the expressions you're looking for and, if it find any, it will output the pieces you want. Just think of as a grip: using grep is a great way to grab a hold of specific lines or files. This command line utility was originally designed for Unix systems.
For example, let's say we have a list of numbers in a file, numlist.txt, and we want to see if the number 8723 is in the list without searching for hours. Using grep can easily find the string in the following command line:
grep –i 8723 numlist.txt
If we get an output of 8723, then it is in the list. If we do not get an output, it is not there. The tool will print the whole line where the text string is found. If the list contained 8723 297r, we would get:
[root@tower root]# grep 8723 numlist.txt 8723 297r
[root@tower root]#
Grep is case sensitive by default. To disable this, use the –i flag.
Here's something useful in a daily RHEL system checkup. Say you need to find out which version of sendmail rpm you installed. To list which rpm packages are installed on the system, you enter the following:
rpm –qa
If you have a system like mine, you have just too many packages to sort through. It might take forever to go through them line by line to find the package you're looking for. Wouldn't it be easier to find a line with a text string matching the word "sendmail?" Thankfully, we can use grep like this:
rpm –qa | grep sendmail
Which should give you:
[root@tower root]# rpm -qa | grep sendmail sendmail-8.13.1-3.RHEL4.5
[root@tower root]#
Don't know if the string is capitalized or not? Just turn off the case-sensitive default:
rpm –qa | grep –i sendmail
Here's another example of how handy grep is. If we want some information on a process that is running on our server, we would run the command ps, which returns a fairly large list of running processes. To find the certain process that we're looking for, put the grep command after ps. Say I need to make sure my DHCP Server is running. I'll run the following:
ps ax | grep –i dhcp
You should see something like the following:
3058 ? Ss 0:00 /usr/sbin/dhcpd -cf /etc/dhcp/dhcpd.conf -q -pf /var/run/dhcp/dhcpd.pid -user dhcp -group dhcp
All set. Our DHCP server is running thanks to the simple and efficient grep.
Using sed
The command sed (stream editor) is used to take a file, read a line, and change the line as specified by the command. This is another former UNIX utility available today in most Linux distributions.
Let's say we have a file called sedtest.txt, which contains the string "1.0". We want to modify this file, changing the string to "2.0". We can do this by doing the following:
sed -e 's/1.0/2.0/g' sedtest.txt > sedtest2.txt mv –f sedtest2.txt sedtest.txt
In this command line, s stands for substitute and g stands for global. Sedtest.txt is the input file (the one we're changing) and sedtest2.txt is the file we've created from the changes of sedtest.txt. The second command takes the output file and renames it to the input file name (replacing the changes made)
Besides substitution, there are other things sed is useful for. For example, the following deletes empty lines or lines that only contain spaces:
sed -e '/^ *$/d' sedtest1.txt
Notice how this one does not need an output file.
Just like the Replace function in Word and Wordpad, sed is the Linux version of this handy tool.
Monday, October 25, 2010
Creating an inventory with nmap network scanning
Creating a system inventory is a common task for system and network administrators. Information security specialists often need inventories as well. Even in shops that have inventory tools, a method is often needed to validate the inventory and identify systems that the tool may not be aware of.
The open source security audit tool nmap has all the necessary features needed to run ad-hoc or automated inventory processes. However, it is not obvious which parameters and output format are ideal to quickly create an inventory of your environment in a useable format.
Generating usable nmap results
To create a usable inventory, the format of the generated data is crucial. If a report is being shared, a portable format is likely needed. Comma-separated values (CSV) are ideal, as this format can be loaded easily into spreadsheet and database programs. If the report is not shared, then a format that stands on its own must be included with the output. This article will provide example of the CSV generation.
The inventory created by nmap and other network polling tools is a network-based inventory. The inventory created provides information that is critical to system, application and protocol management, such as a system's IP address, its operating system and the applications that it is running on network ports. The inventory will not include information such as how much memory the system has, how many processors there are, etc. This type of hardware inventory requires either an SNMP agent on the system or some other program or script running on the system to determine the hardware in the system.
Creating systems reports
A systems report can be quickly generated by nmap. Simply by scanning a network, an immediate list of systems and their protocols can be seen by using operating system identification (-O) and possibly verbose output (-v) against the complete network (/24):
nmap -O -v 192.168.1.0/24
However, the format is unruly, as pages and pages of output are generated. What is needed is a very clean output that can be easily loaded into a spreadsheet.
Nmap supports the output parameter (-o) to influence how it should write data to standard out. By using it combined with G (-oG), nmap will create output that grep can work easily with, which makes our inventory creation much easier.
Using operating system identification and the “grepable” output formatting, the following command can be used to run the raw reports and output the report to report.txt:
nmap -O -oG report.txt 192.168.1.0/24
This report, however, is still difficult to read and is not easily readable by a spreadsheet program or database, and includes the IP addresses that are not assigned to a system, since it scans the complete network:
cat report.txt
# nmap 5.21 scan initiated Thu Sep 30 08:28:31 2010 as: nmap -O -oG report.txt 192.168.1.0/24
Host: 192.168.1.0 () Status: Down
Host: 192.168.1.3 () Status: Down
Host: 192.168.1.4 () Status: Down
Host: 192.168.1.5 () Status: Down
Host: 192.168.1.1 (router.domain.com) Status: Up
Host: 192.168.1.1 (router.domain.com) Ports: 80/open/tcp//http///, 443/open/tcp//https///, 4567/open/tcp//unknown///, 8080/open/tcp//http-proxy///, 8443/open/tcp//https-alt/// Ignored State:closed (995) OS: Linux 2.4.18 - 2.4.35 (likely embedded) Seq Index: 205 IP ID Seq: All zeros
…
To create a system report that includes IP address, hostname and operating system, we need to focus on the lines containing the information. The easiest way to identify these lines is to search for OS:
grep "OS:" report.txt
Host: 192.168.1.1 (router.domain.com) Ports: 80/open/tcp//http///, 443/open/tcp//https///, 4567/open/tcp//unknown///, 8080/open/tcp//http-proxy///, 8443/open/tcp//https-alt/// Ignored State: closed (995) OS: Linux 2.4.18 - 2.4.35 (likely embedded) Seq Index: 205 IP ID Seq: All zeros
…
We now have the information necessary to create the report. However, a lot of information exists in the entry (such as open ports, sequences, etc.) that is not needed and should be dropped during the report generation. Let's remove Host:, Ports:, and all the ports included; OS:, everything starting with Seq up to the end of the line; and, finally, the first set of parenthesis around the host name:
grep "OS:" report.txt | sed 's/Host: //' | sed 's/Ports.*OS://' | sed 's/Seq.*$//' | sed 's/(//' | sed 's/)//'
The ouput of this command is in this format:
192.168.1.1 router.domain.com Linux 2.4.18 - 2.4.35 (likely embedded)
Now the IP address, hostname and operating system are easily identifiable, and it's a matter of piping that output to awk to add quotes and commas for the CSV. Basically, quotes are added around the IP address, the host name and all the words that make up the operating system identification tag:
grep "OS:" report.txt | sed 's/Host: //' | sed 's/Ports.*OS://' | sed 's/Seq.*$//' | sed 's/(//' | sed 's/)//' | awk '{print "\"" $1 "\",\""$2"\"," $3 " " $4 " " $5 " " $6 " " $7 " " $8 " " $9 " " $10 " " $11 " " $12 " " $13 " " $14 "\""}' >report.csv
Looking at the newly created report.csv file, the output is now in CSV format and can easily be loaded into a spreadsheet:
cat report.csv
"192.168.1.1","router.domain.com","Linux 2.4.18 - 2.4.35 (likely embedded) "
"192.168.1.2","freddy","Linux 2.4.35"
"192.168.1.6","computer.home","Linux 2.4.21 (embedded)|MontaVista embedded Linux 2.4.17 "
"192.168.1.7","IMAC.home","Apple Mac OS X 10.5 - 10.6.1 (Leopard - Snow Leopard) (Darwin"
"192.168.1.8","new-host-3.home","Apple iPhone mobile phone (iPhone OS 3.0 - 3.2, Darwin 10.0.0d3) "
"192.168.1.11","new-host-5.home","Apple Mac OS X 10.5 - 10.6 (Leopard - Snow Leopard)"
"192.168.1.15","AppleTV.home","Apple Mac OS X 10.4.8 - 10.4.11 (Tiger) (Darwin 8.8.0 – 8.11.1)"
Conclusion
Using the output of nmap, combined with the power of grep, sed, and awk, a complete network inventory can be generated in a matter of seconds on a small environment, and in a matter of minutes even in a very large environment.
These same tools can be used by the information security specialist seeking to identify changes such as additions and removal of systems within the environment.
The open source security audit tool nmap has all the necessary features needed to run ad-hoc or automated inventory processes. However, it is not obvious which parameters and output format are ideal to quickly create an inventory of your environment in a useable format.
Generating usable nmap results
To create a usable inventory, the format of the generated data is crucial. If a report is being shared, a portable format is likely needed. Comma-separated values (CSV) are ideal, as this format can be loaded easily into spreadsheet and database programs. If the report is not shared, then a format that stands on its own must be included with the output. This article will provide example of the CSV generation.
The inventory created by nmap and other network polling tools is a network-based inventory. The inventory created provides information that is critical to system, application and protocol management, such as a system's IP address, its operating system and the applications that it is running on network ports. The inventory will not include information such as how much memory the system has, how many processors there are, etc. This type of hardware inventory requires either an SNMP agent on the system or some other program or script running on the system to determine the hardware in the system.
Creating systems reports
A systems report can be quickly generated by nmap. Simply by scanning a network, an immediate list of systems and their protocols can be seen by using operating system identification (-O) and possibly verbose output (-v) against the complete network (/24):
nmap -O -v 192.168.1.0/24
However, the format is unruly, as pages and pages of output are generated. What is needed is a very clean output that can be easily loaded into a spreadsheet.
Nmap supports the output parameter (-o) to influence how it should write data to standard out. By using it combined with G (-oG), nmap will create output that grep can work easily with, which makes our inventory creation much easier.
Using operating system identification and the “grepable” output formatting, the following command can be used to run the raw reports and output the report to report.txt:
nmap -O -oG report.txt 192.168.1.0/24
This report, however, is still difficult to read and is not easily readable by a spreadsheet program or database, and includes the IP addresses that are not assigned to a system, since it scans the complete network:
cat report.txt
# nmap 5.21 scan initiated Thu Sep 30 08:28:31 2010 as: nmap -O -oG report.txt 192.168.1.0/24
Host: 192.168.1.0 () Status: Down
Host: 192.168.1.3 () Status: Down
Host: 192.168.1.4 () Status: Down
Host: 192.168.1.5 () Status: Down
Host: 192.168.1.1 (router.domain.com) Status: Up
Host: 192.168.1.1 (router.domain.com) Ports: 80/open/tcp//http///, 443/open/tcp//https///, 4567/open/tcp//unknown///, 8080/open/tcp//http-proxy///, 8443/open/tcp//https-alt/// Ignored State:closed (995) OS: Linux 2.4.18 - 2.4.35 (likely embedded) Seq Index: 205 IP ID Seq: All zeros
…
To create a system report that includes IP address, hostname and operating system, we need to focus on the lines containing the information. The easiest way to identify these lines is to search for OS:
grep "OS:" report.txt
Host: 192.168.1.1 (router.domain.com) Ports: 80/open/tcp//http///, 443/open/tcp//https///, 4567/open/tcp//unknown///, 8080/open/tcp//http-proxy///, 8443/open/tcp//https-alt/// Ignored State: closed (995) OS: Linux 2.4.18 - 2.4.35 (likely embedded) Seq Index: 205 IP ID Seq: All zeros
…
We now have the information necessary to create the report. However, a lot of information exists in the entry (such as open ports, sequences, etc.) that is not needed and should be dropped during the report generation. Let's remove Host:, Ports:, and all the ports included; OS:, everything starting with Seq up to the end of the line; and, finally, the first set of parenthesis around the host name:
grep "OS:" report.txt | sed 's/Host: //' | sed 's/Ports.*OS://' | sed 's/Seq.*$//' | sed 's/(//' | sed 's/)//'
The ouput of this command is in this format:
192.168.1.1 router.domain.com Linux 2.4.18 - 2.4.35 (likely embedded)
Now the IP address, hostname and operating system are easily identifiable, and it's a matter of piping that output to awk to add quotes and commas for the CSV. Basically, quotes are added around the IP address, the host name and all the words that make up the operating system identification tag:
grep "OS:" report.txt | sed 's/Host: //' | sed 's/Ports.*OS://' | sed 's/Seq.*$//' | sed 's/(//' | sed 's/)//' | awk '{print "\"" $1 "\",\""$2"\"," $3 " " $4 " " $5 " " $6 " " $7 " " $8 " " $9 " " $10 " " $11 " " $12 " " $13 " " $14 "\""}' >report.csv
Looking at the newly created report.csv file, the output is now in CSV format and can easily be loaded into a spreadsheet:
cat report.csv
"192.168.1.1","router.domain.com","Linux 2.4.18 - 2.4.35 (likely embedded) "
"192.168.1.2","freddy","Linux 2.4.35"
"192.168.1.6","computer.home","Linux 2.4.21 (embedded)|MontaVista embedded Linux 2.4.17 "
"192.168.1.7","IMAC.home","Apple Mac OS X 10.5 - 10.6.1 (Leopard - Snow Leopard) (Darwin"
"192.168.1.8","new-host-3.home","Apple iPhone mobile phone (iPhone OS 3.0 - 3.2, Darwin 10.0.0d3) "
"192.168.1.11","new-host-5.home","Apple Mac OS X 10.5 - 10.6 (Leopard - Snow Leopard)"
"192.168.1.15","AppleTV.home","Apple Mac OS X 10.4.8 - 10.4.11 (Tiger) (Darwin 8.8.0 – 8.11.1)"
Conclusion
Using the output of nmap, combined with the power of grep, sed, and awk, a complete network inventory can be generated in a matter of seconds on a small environment, and in a matter of minutes even in a very large environment.
These same tools can be used by the information security specialist seeking to identify changes such as additions and removal of systems within the environment.
Subscribe to:
Posts (Atom)