22 thoughts on - C7: How To Configure Raid At Install Time

• –Don’t feel bad. The abortion that is the RHEL/CentOS 7 graphical install interface is far too dumbed-down to be easily usable by anyone that understands what is going on under the covers. Oh, the irony.

  A quick google brought up the following link that (looking just at the disk portion) appears to be mostly correct, and should give you the magic incantation:

  <http://www.ictdude.com/howto/install-CentOS-7-software-raid-lvm/>

  The one thing I would point out regarding the above link is that despite conventional UNIX wisdom, *don’t* put /usr on a separate filesystem in CentOS 7. Thank you RedHat

  Flames to /dev/null.

  Devin

• that looks great, thank you!

• Fred Smith wrote:

  Agreed. What I do, all the time for a new server, is to get to the screen in the graphical install where you select a drive, then , and do what I need to – partition, make a RAID, whatever, then switch back to the graphical, and now set mount points, etc.

  mark

• Fred Smith wrote:
  
  Right, hit , then thought of one more annoyance: the stupid graphical custom partition option will NOT do what you tell it: I want
  /boot first, then swap, *then* /, but if you don’t do it the way I
  mentioned in my last post – and parted, it renumbers and reassigns your partitions Magically , and puts / after /boot, and before swap. For no particular reason, and with no explanation.

  mark

• Indeed, make something that even idiot can use, and only idiot will [be happy to] use it.

  My C7 boxes (workstations, no C7 servers, please) are all installed using kickstart, and drives are partitioned exactly as I described in kickstart file. There is still something we can happily use ;-)

  Valeri

  ++++++++++++++++++++++++++++++++++++++++
  Valeri Galtsev Sr System Administrator Department of Astronomy and Astrophysics Kavli Institute for Cosmological Physics University of Chicago Phone: 773-702-4247
  ++++++++++++++++++++++++++++++++++++++++

• Sorry, you don’t get to throw that grenade and then run away.

  The old wisdom you refer to is simply obsolete, and it wasn’t Red Hat that made it so. ’Twas *progress* that made it so, specifically the fact that even a throwaway USB key has enough space to hold the complete OS on it these days. We no longer live in a world of 5 MB disk packs the size of extra large pizzas.

  Several OSes made /usr/{bin,lib} the same as /{bin,lib} way before Red Hat: Solaris, OS X, and Cygwin, at the least. Probably all of the embedded Linuxes, too.

  Even FreeBSD is starting to give up on /usr as separate from /. Although its installer still lets you put /usr into a separate slice, the boot process will break if you put it onto a different physical disk that the boot kernel can’t see, or use a different filesystem for it that isn’t compiled into the boot kernel.

  A truly “traditional” Unix OS wouldn’t have this problem: as long as the tools necessary to mount /usr are in /, it would be able to boot.

  If you disbelieve that, try installing FreeBSD 9.0 (i.e. pre-ZFS-boot) into a VM, then move /usr into a ZFS pool. It won’t even boot into single-user mode! Ask me how I know.

• Small correction: OS X doesn’t symlink /{bin,lib} to /usr/{bin,lib} as Solaris, EL7, and Cygwin do, but the installer doesn’t offer the option to put it onto a separate volume/partition, and I doubt the OS would boot if you did later move it.

• –For the record, size of the disk was never the original motivation for keeping / separate, at least within my memory; it was to minimize the amount of disk space that needed to be fsck’d before bringing the system to single user mode without needing to revert to “rescue media” (at that time, the 125MB tape on which the OS was shipped). Even in the days of SunOS 3 (that’s SunOS, not Solaris) I was installing the entire OS on one physical drive, partitioned. Sure, NFS and USENET servers had many drives, but most machines were single drives.

  Beyond that, I don’t really want to get into a flame war on stuff that has been hashed through before.

  The main message is that while CentOS 6 and before *could* have
  / and /usr on different filesystems, don’t do it with CentOS 7
  per <https://access.redhat.com/solutions/53005> (paywalled). TL;DR: you’ll have trouble booting some configurations including those using iSCSI.

  Devin

• Not only. Separation of / /var /usr /home (/var/mail, …) also was a protection from local user [accidental] denial of service attack by preventing him from filling up or running out of inodes these filesystems.

  Valeri

  ++++++++++++++++++++++++++++++++++++++++
  Valeri Galtsev Sr System Administrator Department of Astronomy and Astrophysics Kavli Institute for Cosmological Physics University of Chicago Phone: 773-702-4247
  ++++++++++++++++++++++++++++++++++++++++

• Prior to SysV, the location of user home directories was not standardized, and AT&T recommended that you put them in /usr.[1]

  Also, in the PDP days, you had things like the RL and RK series drives, which started at 2.5 MB. If you were well-heeled, you might have several of these, getting you up into the tens of megs. Thus, it made sense to put one major filesystem on each disk, like /, /usr, and /var for a 3-drive system.

  That may have been a happy side benefit, but I’ve never read such a thing in a quarter century of using Unix.

  Reference?

  Unix goes back about 15 years before the advent of the Sun-3. Being fairly high-end boxes of the time, they likely would have had HDDs in the hundreds of megs range, possibly even the single-digit gigs, plenty for a complete 4.2BSD install.

  This was also the time in computing history where diskless or small-disk workstations were common, which gives another reason to make /usr separate: get an NFS stack up via RARP or the small local system drive, then attach to a server’s disk to complete the boot process.

  I don’t need to pierce the paywall to know it’s a bad idea. This is enough of a clue:

  $ ls -l /bin
  lrwxrwxrwx. 1 root root 7 Jul 7 2014 /bin -> usr/bin

  [1]: See page 4-8 in the SVR3.2 sysadmin’s guide: http://goo.gl/E9quko

• Sorry, that should be “SVR4”.

• One of the things I’m working on right now is moving toward a single standard partition layout for all systems, with RAID or without. The reason I want to do that is that it’ll allow me to script the replacement of a failed drive, both so that I don’t fat-finger it and so that our on-call staff can reasonably expect to replace a drive.

  In order to do that, I set up GPT on all drives, regardless of whether they are used in BIOS systems or UEFI. On BIOS systems, the first partition is a bios_boot partition, and on UEFI, it’s an EFI system partition. The next partition is a small slice for RAID1 /boot. The last partition is a member of whatever RAID level is appropriate for the system.

  There are a couple of limiting factors on that. First, Anaconda won’t create a large partition for bios_boot. Second, even in kickstart, Anaconda will reorder partitions. So, I can’t use Anaconda to create partitions and achieve a uniform layout.

  I use this kickstart file for initial setup:
  http://home.dragonsdawn.net/kickstart/CentOS7

  And this script to replace drives:
  http://home.dragonsdawn.net/kickstart/add-raid-member

  I still haven’t decided how I’m going to handle EFI system partitions in the replacement script and otherwise, so this system is incomplete. But if people want to take a look and comment, I’d be interested in discussion.

• I really don’t think that’s an appropriate or useful way to discuss software.

• Perhaps by luck. Anaconda will definitely reorder partitions if you describe them with the “part” command in kickstart. The *only* way to get predictable partition layouts is to manually partition.

• And indeed it did. Not to the extent of going ultimately against my scheme. OK, CentOS (7) just fell yet one more notch down in my opinion. Thanks for correcting me.

  Valeri

  ++++++++++++++++++++++++++++++++++++++++
  Valeri Galtsev Sr System Administrator Department of Astronomy and Astrophysics Kavli Institute for Cosmological Physics University of Chicago Phone: 773-702-4247
  ++++++++++++++++++++++++++++++++++++++++

• Have a look at the links I sent earlier and see if that approach might work for you. If not, I’d be interested in what needs it doesn’t meet.

• I’ve been building C7 VMs with RAID-1 in Virtualbox these last couple of days, to pin down exactly how to do it. based on the link above, but I’m making “real” partitions instead of LVM.

  The current iteration seems to be running fine, but I had an odd problem when configuring the partitions: for two 10 gig virtual drives, it wouldn’t let me use the last gig of space. or maybe it has some other issue on the max partition size,… it seemed to max out at around 9900 MB. This for sure won’t do when I build it on real hardware, so I’m wondering if anyone else here has a clue what’s going on??

  thanks in advance!

• IIRC, Anaconda can’t create “raid1” LVM volumes. That is, you can’t create LVs that are mirrored across multiple PVs. You can create md RAID1 volumes and make a PV of the resulting RAID volume, though.

  So I think what you’re probably doing is creating 100MB /boot partition, and then two md RAID component partitions of 9900MB.

• yeah, you’re probably right. but for some reason, Anaconda is still telling me there’s a gigabyte of unused space remaining (105x megabytes where i’ve forgotten what the “x” was.) Might that be because I
  didn’t put swap on both drives (and I think I set up swap as a gig)?

  should I put swap on both drives? should it be RAID1 also, or just two swap partitions not in any RAID volume?

  (I remember seeing a somewhat technical article in the last couple of years that talked of a potential corruption issue if one puts swap into a Linux software RAID-1 volume, but no longer recall the details.)

• I really recommend using the fewest partitions possible, since md RAID
  operates on partitions (Anaconda won’t create full disk md RAID
  volumes), and replacing a disk will require you to handle each partition individually. It’s probably best to do a small /boot on RAID1 and use the rest of the disk for a second RAID1 volume, with LVM on that.

  (If you haven’t looked at the links I posted earlier, do consider that)

  There’s no corruption. If a write is canceled before it completes, you might end up with a block mis-match (or might have, I’m not sure if that still happens), but no corruption. The mis-match is on a block that isn’t mapped to anything, and will never be read by anything other than a RAID verify. It’s ugly because it means the verify operation is no longer useful, but it won’t cause any read from the RAID volume to return bad data.

• Fred Smith wrote:
  

  Two thoughts: one, as Gordon suggested, it might be making a /boot partition. The other thought is that it might be trying to align the partition optimal. Still, that’s usually 1M or 2M, not 1G.

  Btw, when I manually create partitions, with parted, I use mkpart pri 0.0GB 100% (for data drives). Assuming I’ve gone in with parted
  -a opt, it does the right thing. Hope this gives you some ideas.

  mark

• This is not a large burden, but I do agree to keep it simple.

  The default configuration is a partition for /boot and one per drive for LVM PVs all of which are used by a single VG, in which LVs for / (which has a fixed upper size), swap (which uses a computed size) and if there’s space available /home (which uses all remaining space).

  And there’s the problem, the installer uses all available space leaving none for a RAID of /boot/.

  I believe this is due to an assumption that Red Hat makes, that their customers are mostly enterprise and thus the drives that the installer sees would be logical and backed by RAID.

  I could wish that the installer would RAID /boot by default, which is small enough that it is unlikely to be a burden for those with RAID
  backed “drives” and a boon to those who do not.

  What’s needed is enough space for the installer to carve out additional partitions on the additional drives. Which means manual partitioning, and in particular you must delete or resize /home — remember, this is in the initial configuration so there’s nothing to lose, and you can create/resize it a little later anyway. If you have very small disks you probably need to shrink / and/or swap.

  Reduce the space to be used by enough for another /boot and you can change /boot’s device type to one of the RAIDs.

  Reduce it to no more than what the first drive provides and you can also setup RAID under the PV — modify the volume group redundancy.

  After which you can make mount points for anything else, e.g., /var/www, and grow what you shrank.

  If you like pain you can back into such a layout after installation is complete, but I do not recommend it.

  When there are more than 2 drives possible I prefer MD RAID for /boot and LVM RAID for everything else — the not likeable part is that even though LVM RAID leverages MD RAID the status is not presented via
  /proc/mdstat, which changes how one must monitor for failures. When only 2 drives are possible odds are you want RAID1 and it might as well be under the PV.

  Using LVM RAID is interesting in that it allows you to decide later without much effort the redundancy you want on a per mount point basis, e.g., on a 4 drive system it is just two commands to create mount points
  (LVs) using RAID10 for /var/lib/*sql but RAID5 for /var/www — that’s not impossible to do with MD RAID under PVs but it probably means lots more commands and even some juggling.

  /mark