Software RAID on Linux with mdadm

RAID Controllers

We’ve spent a great deal of time examining various RAID levels such as RAID-0, 1, 5, and 6, and Nested RAID levels such as RAID-10, 50, 51, 61, and 60 or even the more complicated RAID-100 or RAID-160. In all of these discussions we have assumed there was a RAID “controller” that performed the various RAID operations. This includes sending chunks of data to the appropriate disks, computing parity, hot-swapping, disk fail-over, checking read transactions to determine if the read was successfully and if not, declaring that disk as “down”, plus other important tasks related to RAID. All of these tasks require some sort of computation and have to be performed by a RAID controller.

You really have two options for RAID controllers: (1) a dedicated RAID controller such as those on add-in RAID cards, (2) software RAID that uses the CPU for RAID chores. In the first case you have a dedicated RAID controller, typically on an add-in RAID card but it can be on a motherboard, that performs the necessary RAID computations. This controller typically uses a dedicated lower power processor, often a real-time processor such as a PowerPC, to perform the computations. Typically these controllers are put on an add-in card and you plug the drives you want in the RAID array into the card.

With software RAID, all the RAID functions run on the CPU. Pretty much all Linux distributions comes with software RAID in the form of md (Multiple Device) which is a driver within Linux that provides virtual devices created from one or more underlying devices (e.g. storage block devices). Plus the great thing is that it comes with almost all Linux distributions so you have access to the source (plus the price isn’t too bad either and it has a great deal of functionality). In addition, you can create Nested RAID configurations as needed since all RAID functions are in software (I smell a Triple Lindy coming).

One thing you have to be very careful about is what is commonly called a fakeRAID card or fakeRAID controller. FakeRAID is not hardware RAID because there is no dedicated RAID controller. Rather, they use a standard disk controller chip on an add-in card or motherboard, with some specialized firmware and drivers. At boot-time they run a special application that allows users to configure disks attached to the fakeRAID as a RAID group. But the RAID processing is really handled by the drivers which are run on the CPUs (so the CPU provides the computational power). Consequently, it’s really a software RAID solution and not hardware RAID (hence the name “fakeRAID”).

There is a great deal of discussion and grinding of teeth within the Linux community about fakeRAID. One point of the discussions is that the vendors of fakeRAID don’t tell customers that what they are actually buying is not a RAID card with a dedicated RAID controller, but rather a simple card with a disk controller coupled with drivers that use the CPU for RAID processing (false advertising). Plus there is the additional problem of developing and supporting drivers for Linux to allow these fakeRAID cards to be used. Moreover, there is a strong argument that it is probably better to use software RAID that comes with Linux (md) since it is already part of Linux and can arguably give you better performance.

However, if you want to use software RAID that comes with Linux in the kernel (md), you still need some tools to control/manage/monitor the software RAID arrays. That’s where mdadm comes in. This article will do a brief examination of mdadm and some of its options.

Introduction to mdadm

Mdadm is a software tool primarily written by Neil Brown that allows you to create, assemble, report on, grow, and monitor RAID arrays that use software RAID in Linux. Actually according to the documentation there are seven modes of operation:


   1. Create
   2. Assemble
   3. Follow or Monitor
   4. Build
   5. Grow
   6. Manage
   7. Misc


We’ll walk through these various modes of operation but the focus of this article is an introduction and not an in-depth HOWTO. You can find those types of articles on the web.

The first step in using mdadm, or any RAID for that matter, is to PLAN your RAID configuration carefully. Personally I like to work backwards starting with the purpose of the storage. Will it be used for a database? Will it be used for /home? Will it be used for high-speed scratch? Will it be used for data that requires a very high degree of reliability? Understanding the intent of the storage is really the key to creating the RAID configuration you need/want. Once you determine the storage use case, you need to develop an idea of how much I/O performance you need (throughput and IOPS) and the general ratio of read and write performance. You should also develop an idea of how much data redundancy you need for the storage.

Once you have an idea of the performance and redundancy of the array you can select the RAID configuration you think you might need. I would suggest you select a few candidate RAID configurations and then do some more reading/research on each one and select the RAID configuration that seems to be best. During this research be sure to examine the redundancy as well as the performance of the various RAID configurations and compare them to your estimations. But also be sure to examine the capacity and storage efficiency of each level. You may love the performance and storage efficiency of RAID-10 but the data redundancy may not be enough for you. Or you may love the data redundancy of RAID-61 but you may not willing to give up the performance or, perhaps more importantly, you may not be willing to have such low storage efficiency (especially if this is for your home system).

But just choosing the RAID configuration you want is not the end of your planning. You need to also consider a number of other things. Perhaps the most important thing you need to consider is if you will need to grow/shrink the storage. The reason this is important is because you are likely to have to use LVM (Logical Volume Manager) either on top of Linux software RAID or underneath it. This forces you to carefully consider how to build both LVM and software RAID and how you expand either one or both. I would recommend walking through the expansion steps to make sure you understand how to do it (you could even pass along your ideas to someone else to have another pair of eyes examine them).

One other thing you should consider before implementing your well formulated and thought-out RAID plan is the file system that sits on top of the storage. Based on your usage model for the storage, select one or two candidate file systems. Then do some research on each one to find out what problems or limitations exist, and also how you can optimize each file system for better performance (we’re all performance junkies at heart). There are a number of articles on the web that discuss tuning file systems with Linux software RAID.

Assuming that you have done your careful planning (including a backup solution), let’s move on to the first “mode” of mdadm, Creating a RAID array.

Creating a RAID array

Mdadm allows you to create a RAID array using Linux block devices. During the creation of the array, per-device superblocks are created for the RAID array (allows for the array to be assembled correctly). Using the “create” mode is the most common method for building the array and is recommended if you are just starting to use mdadm.

The basic mdadm command for creating a RAID array is fairly simple with the following generic command and typical options.

mdadm --create [md-device] --chunk=X --level=Y --raid-devices=Z [devices]


where the options are as follows:


      -c, –chunk= Specify chunk size in kibibytes. The default is 64.


      -l, –level= Set raid level, options are: linear, raid0, 0, stripe, raid1, 1, mirror, raid4, 4, raid5, 5, raid6, 6, raid10, 10, multipath, mp, faulty


      -n, –raid-devices= Specify the number of active devices in the array.

Notice that the mdadm command begins with the “–create” option that tells mdadm that it will operate in “create” mode. I like to then define the specific md-device as well but be sure you aren’t using an existing md-device name. Specifying the “chunk” option is up to you. Then you define the RAID level you want by the “–level” option with the options listed above. You can also tell mdadm how many block devices you are using with the “–raid-devices=Z” option (Z is the number of devices). Then finally you give mdadm the list of block devices you are using.

An example of using the create option with mdadm is,

% mdadm --create --verbose /dev/md0 --level=0 --raid-devices=3 /dev/sda1 /dev/sdb1 /dev/sdc1


which creates a RAID-0 configuration that is labeled as /dev/md0 and uses three block devices that are /dev/sda1, /dev/sdb1, and /dev/sdc1.

In the example, I have used the first partition of each of the three drives that are valid block devices as the block devices for mdadm. They could have easily been the entire disk such as /dev/sda or /dev/sdb. The point is that they need to be valid Linux block devices (they could even be network based devices but that’s a different discussion).

Mdadm is smart enough to build the RAID-0 configuration using the smallest common size of each of the three devices. So it is recommended that you check on the size of each block device using fdisk as shown below.

root@amadys-laptop:~/# /sbin/fdisk /dev/sdb

The number of cylinders for this disk is set to 19457.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
   (e.g., DOS FDISK, OS/2 FDISK)

Command (m for help): p

Disk /dev/sdb: 160.0 GB, 160041885696 bytes
255 heads, 63 sectors/track, 19457 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x000bca3e


   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1   *           1       18704   150239848+  83  Linux
/dev/sdb2           18705       19457     6048472+   5  Extended
/dev/sdb5           18705       19457     6048441   82  Linux swap / Solaris


Be sure to look at the column labeled “Blocks” for the particular partition you are going to use. Do this for all the devices and make sure the number of blocks is the same or very close (otherwise you are wasting space).

You might also notice that I used the “–verbose” option in the mdadm create command. I like to use this option to get more information about what mdadm is doing (“better informed than sorry” is a good motto). This is always a good habit to develop.

At this point, your RAID array should be created and running. An easy way to check this is to look at /proc/mdstat.

% cat /proc/mdstat


Fortunately, the output should be fairly easy to read at first glance. For much more detailed information you can read this article.

Immediately after the RAID array is created it may have to go through a synchronization process. This process performs the necessary RAID functions for configuration you created. For example, for RAID-1, the blocks on the first drive are copied to the second drive even if there isn’t any information on the blocks.

Once the array has finished synchronizing and is ready, then you can move to the next step which can be using the resulting RAID array device in LVM or creating a file system using the block device.

There are many options that you can use in “create” mode. You can read the man pages to get a list of them but below are some of the more important ones that haven’t been presented yet in this article.


      -x, –spare-devices=  This option allows you to specify spare devices in the initial array. These are devices (disks) that are used in the event that a disk in the RAID configuration fails. Mdadm then uses the spare drive immediately and starts restoring the array to the desired configuration. Mdadm also allows spare drives to be added and removed later (they don’t have to be added when the array is created). If you use spare devices be sure that the “–raid-devices” option lists the number of devices to be the actual RAID drives plus the spares.


      -p, –layout=, –parity= Mdadm gives you remarkable control over your RAID configuration. This option lets you control the fine details of the data layout for RAID-5 and RAID-10 arrays and also controls the failure mode for a faulty or failed disk. Please read the manpages for more detail.


      -z, –size= This option is the amount of space to be used from each drive in a RAID-1, RAID-4, RAID-5, or RAID-6 configuration. The size is given in kibibytes and must be a multiple of the chunk size. In addition, you must leave about 128KB (128 kibibytes) of space at the end of the drive for the RAID superblock. After the array is created you can use the “grow” mode of mdadm (–grow) to increase the size of the RAID configuration.


Assembling an mdadm RAID array

One of the other “modes” in mdadm is “assemble”. After your RAID array has been created using mdadm, you can stop the array using the following command:

% mdadm --stop /dev/md0


which stops the RAID array /dev/md0 (be sure to unmount the file system that uses the RAID array first). However, there are problems in restarting the RAID array. When you restart the array you have to use mdadm to reassemble the array. For example,

% mdadm --assemble /dev/md0 /dev/sda1 /dev/sdb1 /dev/sdc1


This command assembles the parts of a previously created array into an active array and “restarts” the array (i.e. make it function). To automate this, you could put this command as part of the system startup (for example, /etc/rc.d/rc.local) and you could create a simple script for stopping and starting the array. But mdadm can do some of the leg work for you with the following command:

% mdadm --assemble --scan


These options allow mdadm to scan the drives and reassemble the RAID array (it looks for the RAID superblocks on the drives). Typically this is done during the init phase of the system starting. For example, on my CentOS 5.5 system, there is a line in /etc/rc.d/rc.sysinit that looks like the following.

/sbin/mdadm -A -s


which will scan (“-s”) the drives and assemble (“-A”) the mdadm arrays.

However, you can get into trouble with scanning and assembling mdadm RAID arrays when you have more than one array. What is recommended is that if you want to restart an array by hand you specify the uuid for the array (a unique “name” of the RAID array). For example,

% mdadm --scan --assemble --uuid=7121b438:7d36f9f6:8aa9c8b3:b5b0d211


Since the uuid’s are unique to each array, this will ensure that mdadm can reassemble the array properly. However, mdadm‘s scanning and assembling capabilities are quite good and I routinely run two mdadm RAID configurations on my desktop and I’ve never had any confusion between which disks belong to which array (thanks to the superblocks on the devices).

Monitoring/Following an mdadm RAID array

The third mode of operation of mdadm is monitoring or following an mdadm RAID array. This mode monitors one or more arrays and allows action to be taken if the state of the array changes. According to the manpage for mdadm, this mode of operation is really only useful for RAID-1, 4, 5, 6, and 10, or multipath arrays since they have interesting states. On the other hand, RAID-0, and linear RAID are not useful because missing, spare, or failed drives cause these RAID modes to fail (i.e. non-operational).

The basic option for following or monitoring mdadm controlled arrays is the following:

mdadm --monitor options... devices...


(Note: you can use “-F” or “–follow” in place of “–monitor”). There are several options that can be used for monitoring and following mdadm arrays as listed below:


      -m, –mail This options allows you to define an email address where mdadm alerts are sent.


      -p, –program, –alert This option allows mdadm to run a “program” whenever an event is detected (it is recommended to use the full path to the program).

      -y, –syslog This option causes all events to be reported through “syslog”.

      -d, –delay This options creates a delay (in seconds) from when mdadm polls the arrays to when it next polls the arrays (i.e. the interval between polling).

      -f –daemonize This options tells mdadm to run as a background daemon if it is monitoring arrays. This causes mdadm to fork and run in the child process and disconnect from the terminal.

      -i, –pid-file This option tells mdadm to write the pid of the daemon process when mdadm is run as a daemon (see previous option). The pid is written to a specified file.

      -1, –oneshot This option checks the arrays only once and generates “NewArray” events as well as “DegradeArray” and “SparesMissing” events (this will show up in the logs). According to the manpages, if you run the command “mdadm –monitor –scan -1″ from a cron job, it will ensure regular notification of any degraded arrays (always a good thing).

      -t, –test This option generates a “TestMessage” alert for every array found at startup. This alert gets mailed and passed to the alert program (if you have defined one). This is very useful for testing that alert messages get through successfully (i.e. they work).

As you can see, mdadm gives you several pretty good options for monitoring your arrays. The exact practice of how you monitor really depends upon how you want to function and any tools or processes you have developed. There are several articles on the web that show how to monitor your mdadm array.

In the interest of helping interested people get started I will present a few tips that you can use to get started with monitoring mdadm arrays. I first recommend configuring mdadm to email you in the event of a change in state of the array. There are several articles that discuss how to do this. For example, this blog shows you how to use mdadm to send email to you in the event of a problem.


If you want to be more involved in the monitoring of your arrays there are two primary ways to get more detail: (1) cat /proc/mdstat, and (2) mdadm --detail [device]. The first option gives you a quick overview of the status of the array(s). You can parse this output (perhaps with perl or python), and create a special log or send the output to “syslog” to allow processing by syslog tools. Alternatively, you could create a simple monitoring metric that can be used in conjunction with ganglia or something similar.

The second option gives you more detail than mdstat but again, you can parse this output and then perform some action (logs, syslog, ganglia, etc.). But the details are really up to you since you likely have specific processes or techniques for monitoring.

One last option is to use Munin which is a monitoring tool (somewhat similar to Ganglia). It has plugins that allow you to monitor your mdadm created RAID arrays.

Building an mdadm RAID array

Another mode of operation in mdadm is building an array but without having per-device superblocks. This means that you cannot have mdadm scan and assemble the devices into an array. Consequently, you have to be careful that you differentiate between the initial creation of the array and the assembly of the array (or you can lose data). In addition, if you built an array using build any checks that might have been done between devices are not performed. Basically, you have to be very careful when using this mode of operation and know exactly what you are doing.

I really don’t recommend using this mode of operation so I won’t be discussing it. If you are interested in using the “build” then you can read the manpage or you can search around the Internet for references.

Growing an mdadm RAID array

One of the secret weapons you get with mdadm is the ability to grow, reshape, or even change RAID levels with md arrays. There are some limitations for growing and/or reshaping arrays however, just having that ability is a pretty major accomplishment for mdadm.

The basic option for growing or reshaping md arrays is “-G” or “–grow”. There are a number of options that can be used and things can become complex fairly quickly so I won’t go over them in this introductory article. Please read the manpages for the options or you can Google for information about growing md arrays. However, let’s take a high-level look at what the “–grow” option can do.

We can use the “–grow” option to add a third disk to a two-disk RAID-1 configuration or add another disk to our existing RAID-5 configuration. But we can also use the “grow” option to change RAID levels. For example, we can convert a two-disk RAID-1 md array to a two-disk RAID-5 md array. Direct from the mdadm author’s blog is the list of what changes (reshape) can be done:


    * A RAID-1 array can change the number of devices or change the size of individual devices. A 2 drive RAID-1 can be converted to a 2 drive RAID-5.
    * A RAID-4 can change the number of devices or the size of individual devices. It cannot be converted to RAID-5 yet (though that should be trivial to implement).
    * A RAID-5 can change the number of devices, the size of the individual devices, the chunk size and the layout. A 2 drive RAID-5 can be converted to RAID-1, and a 3 or more drive RAID-5 can be converted to RAID-6.
    * A RAID-6 can change the number of devices, the size of the individual devices, the chunk size and the layout. And RAID-6 can be converted to RAID-5 by first changing the layout to be similar to RAID-5, then changing the level.
    * A LINEAR array can have a device added to it which will simply increase its size.
    * RAID-10 and RAID-0: These arrays cannot be reshaped at all at present.


So you can see that there is a great deal of flexibility in mdadm with respect to changing shape, RAID levels, adding devices, removing devices, etc., to md arrays.

As I mentioned previously, the details of reshaping and changing RAID levels can get complicated. I suggest that before you use this option, you read all the literature you can. Then I would ask some experts who can tell you if your commands are correct or not. And before starting anything, I would definitely make sure you have a backup copy of the data and make sure the storage array isn’t in production.

Managing a md RAID array

Managing a md RAID array primarily consists of managing the devices within the arrays. This can include adding, removing, or failing disks within an array.

An easy (sort of) way to tell if you are using the manage mode, is that if you give a device before any options on the mdadm command line, or if the first option is “–add”, “–fail”, or “–remove”, then you are using the manage mode of mdadm. The general form of the mdadm command is,

mdadm device options ... devices ...


The options used in manage mode are:


      -a, –add [device] This option allows you to add the specified devices to the specified array while the array is running.


      –re-add [device] This options allows you to re-add a device to an array that was recently removed from the array.


      -r, –remove [device] This option allows you to remove the specified device. But the device must not be active so it must be either failed or a spare device.


      -f, –fail [device] This option marks the specified device as faulty (failed).


      –set-faulty This option is the same as -f.


You can combine the options in one command but all of the commands must affect the same array.

A simple example of the manage mode of mdadm is,

% /sbin/mdadm /dev/md1 --add /dev/sdc1 --fail /dev/sdb1 --remove /dev/sdb1


In this example, the array, /dev/md1 is the “target” of the mdadm command. The first option, “–add /dev/sdc1″ adds a device (/dev/sdc1) to the array. Then the option “–fail /dev/sdb1″ fails that device telling the array that the device is faulty. Finally, the third option, “–remove /dev/sdb1″ removes the device from the array at which point it can be removed from the system. Note that you need to first fail the device you intend to remove.

One cool feature of mdadm is that if you remove a device (disk) from an array you can add it back (–re-add) to the array and mdadm will only update the changed blocks from when the disk was removed. This is the default behavior if you used superblocks in the creation of the array (i.e. you didn’t use the “build” mode).

Misc

This last mode of operation is sort of a catch-all for options and commands that don’t fit into the other six modes. In general this mode supports some operations on active arrays, operations on component devices, and the gathering of information about the arrays.

The general form of the mdadm command in “misc” mode is the following.

mdadm options ... devices ...


Notice that for a “misc” command, no array was defined before the options. The options for this mode are the following:


      -Q, –query This option examines a device to see if it is an md device and if it is a component of an md array. The information discovered by mdadm is presented in the output.


      -D, –detail [md-device] This option prints out details of one or more md arrays. If you add the options “–brief” or “–scan” the amount of detail in the output is reduced but the format is amenable for /etc/mdadm.conf (an optional configuration file for mdadm).


      -E, –examine This option prints the content of the md superblock on the device (or all devices if no device is specified). As with the “–detail” option, if you use “–brief” or “–scan” the amount of output is reduced and it is more amenable to /etc/mdadm.conf.


      -X, –examine-bitmap This option reports the information about a bitmap file.


      -R, –run This option will start (activate) a partially built md array.


      -S, –stop This option stops (deactivates) an active md array.


      -o, –readonly This option marks the active array as read-only if it is not being currently used.


      -w, –readwrite This option marks the array as readwrite.


      –zero-superblock This option over-writes a valid md superblock in an array. This is useful when using disks (devices) from an old md array are used in a new one.


      -t, –test This option, when used with the –detail option, sets the exit status of mdadm to reflect the status of md device. This can be very useful when scripting monitoring tools. It is also useful if you want start the md array yourself rather than rely on the kernel to autostart it.


Of all of the “misc mode” operations the “–query”, “–detail”, and “–examine” options are the most generally used.

Summary

We’ve spent some time talking about RAID configurations, both single-level and Nested RAID configurations. In all of the discussions we’ve just mentioned that the RAID operations are handled by a “RAID controller”. There are two types of RAID controllers – hardware and software. The hardware RAID controller has a dedicated processor on an add-in card that handles all the RAID computations. In contrast, software RAID uses the system CPU for RAID computations (this includes “fakeRAID” controllers as well).

This article is an introduction to mdadm, the management/admin tool for Linux software RAID that comes with virtually every Linux distribution. The tool is very flexible allowing for the standard RAID levels and Nested RAID configurations, including some specialized RAID-10 configurations we’ve discussed previously. You can even use it to build some “Triple Lindy” Nested-RAID configurations if you want.

Mdadm has seven different “modes” of operation which we discussed. These modes allow you to create and start a RAID array, assemble a RAID array (useful when the system boots), follow or monitor a RAID array, build a RAID array (basically doing everything by hand – not recommended), grow a RAID array (one of the secret weapons of mdadm), manage a RAID array, and a “miscellaneous” category for functions that you may need that didn’t fall into the other categories.

There are some really great features in mdadm that can easily be glossed over in a mad rush to build a RAID configuration. There are two big ones that I want to highlight. The first features is the set of standard monitoring tools in mdadm that give you a great starting place for watching the status of your md array including the ability to send out email alerts. Plus you can write fairly simple scripts to parse array status information which can be used in monitoring tools such as ganglia or munin.

The second feature, which is probably the most significant, is the ability to grow and reshape md arrays. This allows you to add devices to an existing RAID array and grow the array to include the added space. However, the really cool feature is that you can use mdadm to change RAID levels without losing data. For example, you could convert a two-disk RAID-1 into a two-disk RAID-5 configuration. Then you could add disks to grow the RAID-5 configuration. Then you could convert the RAID-5 into a RAID-6 configuration. While I haven’t used this feature, this is pretty nifty if you ask me.

Mdadm is a great tool for Linux that is easy to use and gives you a great deal of control over RAID arrays. If you are thinking about RAID on Linux, be sure to take a look at mdadm.