12 thoughts on - DegradedArray Message

• In short, buy a replacement disk equal or greater size, create matching partitions and then use mdadm to add the replacement partition (of appropriate size) back into the array.

  An example command to add a replacement partition would be:

  mdadm –manage /dev/md0 –add /dev/sda1

  I strongly recommend creating a virtual machine with a pair of virtual disks and simulating the replacement of the drive before trying it out on your real system. In any case, be sure to have good backups
  (immediately).

• Could be a bad drive, as digimer alludes in his reply.

  OTOH, I had a perfectly good drive get kicked out of my RAID-1
  array a fewyears ago just because, well, I guess I could say
  “it felt like it”.

  In reality, I had (in my ignorance) purchased a pair of WD
  drives that aren’t intended to be used in a RAID array, and once in a long while (that was actually the only such instance in the 4-5 years I’ve had that RAID array) it doesn’t respond to some HD command or other and gets dropped.

  turned out to be easy to reinsert it and it ran for a long time thereafter without trouble.

  I can dig for the info on the drives and the nature of the problem if anyone wants to see it.

  Fred

• That looks about right. There may be more error messages in your system logs (e.g., /var/log/messages, dmesg), which might tell you more about the nature of the failure.

  The linux RAID wiki is often a good (though sometimes dated) resource.

  https://raid.wiki.kernel.org/index.php/Reconstruction

  If you wish to attempt a hot swap, make *sure* you pull the correct device! If you’re not sure, or not sure your system even supports it, it’s safer to power down to do the swap. You should verify which drive has failed before shutting down, though this will be less catastrophic if you pick the wrong one.

  As long as you are careful, reconstructing a degraded RAID is usually pretty straightforward.

  –keith

• desktop class SATA drives will report ‘write successful’ when there’s still data in its buffers, so the raid will happily continue, then if the drive actually gets a unrecoverable write error, things are toast, the raid is out of sync.

  this is a major reason I’m leaning towards using ZFS for future raids
  (primarily via using FreeBSD), because ZFS checksums and timestamps every block it writes, so it can look at the two blocks that a regular raid can only say “something is wrong here, but what it is I ain’t exactly sure” and go “A is good, B is bad/stale, lets replicate A back to B”, as part of the zpool ‘scrub’ process.

• I’ve seen that too several times on my home “server”.

  Once in a while (usually on one of the first days I’m on vacation) one of the drives stops responding.

  A shutdown and cold restart is necessary to bring the drive alive again, just a reboot won’t fix it.

  After this, I rebuild the RAID partitions and all is OK.

  smartctl shows no sign of problems with the drive, so I suspect a controller problem.

  This is on a desktop machine used as a server.

  I guess this explains why we have server grade hardware :-)

  Mogens

• I have also seen drives/partitions get kicked of of softraid arrays for unknown reasons. My first choice is to re-add the member into the softraid and see if it 1)
  will re-add and 2) if it stays a member

  As far as re-adding drives to a softraid array, I’ve wrote on this list a few times before. Plus the URLs others have provided will help you as well. Any questions, just ask and somebody will reply.

  http://lists.CentOS.org/pipermail/CentOS/2013-November/138699.html http://lists.CentOS.org/pipermail/CentOS/2014-February/140700.html

  Good luck. Digimer’s advice about testing your scenario out on a VM is perfect one since you’re new to the task.

• Hi David,

  Am 03.12.2014 um 02:14 schrieb David McGuffey :

  the reason why one drive was kicked out (above [_U] ) will be in /var/log/messages. If it is also part of md1 then it should be manually removed from md1 before replacing the hd.

• Thanks for all the responses. A little more digging revealed:

  md0 is made up of two 250G disks on which the OS and a very large /var partions resides for a number of virtual machines.

  md1 is made up of two 2T disks on which /home resides.

  Challenge is that disk 0 of md0 is the problem and it has a 524M /boot partition outside of the raid partition.

  My plan is to back up /home (md1) and at a minimum /etc/libvirt and /var/lib/libvirt (md0) before I do anything else.

  Here are the log entries for ‘raid’

  Dec 1 20:50:15 desk4 kernel: md/raid1:md1: not clean — starting background reconstruction Dec 1 20:50:15 desk4 kernel: md/raid1:md1: active with 2 out of 2
  mirrors Dec 1 20:50:15 desk4 kernel: md/raid1:md0: active with 1 out of 2
  mirrors

  This is a desktop, not a server. We’ve had several short (<20 sec) power outages over the last month. The last one was on 1 Dec. I suspect the sudden loss and restoration of power could have trashed a portion of disk 0 in md0.

  I finally obtained an APC UPS (BX1500G), installed, configured, and tested it. In the future, it will carry me through these short outages.

  I’ll obtain a new 250G (or larger) drive and start rooting around for guidance on how to replace a drive with the MBR and /boot on it.

• …

  Assuming that you have an unused drive port, you can fix that pretty easily.

  Attach a new replacement disk to the unused port. Let’s say that it comes up as /dev/sde.

  Copy the partition table to it (unless it’s GPT, in which case use parted):
  sfdisk -d /dev/sda | sfdisk /dev/sde

  Unmount /boot and copy that partition (assuming that it is sda1):
  umount /boot
  dd if=/dev/sda1 of=/dev/sde1 bs=1M

  Install grub on the new drive:
  grub-install /dev/sde

  At that point, you should be able to also add the new partition to the md array:
  mdadm /dev/md0 /dev/sda2

  Once it rebuilds, shut down. Remove the bad drive. Put the new drive in its place. In theory the system will boot and be whole.

  In practice, however, there’s a bunch of information you didn’t provide, so some of those steps are wrong.

  I’m not sure what dm-0, dm-2 and dm-3 are, but they’re indicated in your mdstat. I’m guessing that you made partitions, and then made LVM or crypto devices, and then did RAID on top of that. If either of those are correct, that’s completely the wrong way to build RAID sets. You risk either bad performance from doing crypto more often than is required, or possibly corruption as a result of LVM not mapping blocks the way you expect.

  If you build software RAID, I really strongly recommend that you keep it as simple as possible. That means a) build sofware RAID sets from raw partitions and b) use as few partitions as possible.

  Typically, I’ll create two partitions on all disks. The first is a small partition for /boot, which may be part of a RAID1 set or may be unused. The second partition covers the rest of the drive and will be used in whatever arrangement is suitable for that system, whether it’s RAID1, RAID5, or RAID10. All of the drives are consistent, so there’s always a place to copy /boot, and just one script or process to set up new disks regardless of their position in the array. md0 is used for
  /boot, and md1 is an LVM PV. All of the filesystems other than /boot are LVs.

  Hopefully btrfs will become the default fs in the near future and all of this will be vastly simplified.

• Gordon,

  Agree, I’ve probably made it too complicated. It is a workstation with sensitive data on it so I’ve encrypted the partitions.

  md1 is fairly simple…two large disks in raid1, encrypted, and mounted as /home.

  md0 is probably way too complicated and not a good way to go. The sensitive data in md0 is in /var (virtual machines).

  I’ve backed up both /home and /var/lib/libvirt/images, so I think I’ll start over on md0 with a new disk and a fresh install.

  Dave

• Armed with a backup I decided to use the disk utility GUI to check the array and then re-attach the disk. After a rebuild phase it reattached and the state changed to ‘clean.’ I rebooted to see if it would stay attached; it did.

  I’ll still get ready for another failure. Will read up on the best methods to have an encrypted filesystem on top of raid-1.

  Dave M

• I’m pretty sure that if you tell the Fedora installer to build an encrypted RAID1 system, you’ll get exactly what I described previously.
  In detail:

  sda1 – 512MB
  sda2 – remainder of disk

  sdb1 – 512MB
  sdb2 – remainder of disk

  md0 – RAID1 including sda1 and sdb1
  md1 – RAID1 including sda2 and sdb2

  /boot – filesystem on md0

  luks-$(uuid) – encrypted block device on md1

  pv.01 – LVM2 physical volume on luks-$(uuid)

  fedora_$(hostname) – LVM2 volume group including “pv.01”

  swap – swap on logical volume root – filesystem on logical volume home – filesystem on logical volume

  If you replace a disk, you’ll need to partition it correctly and “mdadm
  –add” it to the two RAID volumes.

  If you manually set up some other layering, replacing a disk will probably be more involved.