Question About Storage For Virtualisation

Home » CentOS » Question About Storage For Virtualisation
CentOS 20 Comments

I may be behind, but this is the first I’ve heard of this…

Any good references as to WHY?! they want to break this decades old convention?


20 thoughts on - Question About Storage For Virtualisation

  • Am 29.06.2012 20:17, schrieb Nate Duehr:

    Decades old I’ll grant you, but hardly a convention anymore. There was a time when this made sense … about twenty-five years ago. We didn’t have journaling filesystems then, fsck was still new, and if your root filesystem was corrupted you had to boot to single user mode in order to fix it manually. You had every interest to keep your root filesystem small, simple and static.

    But times have changed. Nowadays, a system that has experienced a power cut in full activity will come back up with hardly a hiccup when power is restored. Dividing up your disks into partitions has become a nuisance because there’s always one which is too small while others have space left. So it’s gone out of fashion. I haven’t seen a Unix or Linux system with separate / and /usr partitions in the last five years, and not set one up myself for ten at least.

  • Umm, yeah… Nobody would ever want to boot from small alternate media devices in these days of 3TB drives…. I suppose most of what /bin was supposed to hold and the reasons for putting it there have been usurped by the initrd that is normally required to boot linux, though.

  • Historically for Linux /bin and /sbin were meant to hold the minimum necessary tools to bring the rest of the filesystems online and to do basic restores in case the thing exploded (basically, “single user mode”
    tools). Everything else was meant to be in /usr. /lib requirements came from /bin. At least that was the rationale we put into FSSTND back in the early-mid 90s. (I’m not sure RedHat ever really followed this properly;
    there’s too much stuff in /bin :-))

    With initrd and similar ramdisk boot systems these requirements have changed and the distinction between /bin and /usr/bin has definitely dropped.

  • The concept really comes from the original unix, which back in the day, often had really tiny boot disks and might mount everything else over the network or use different drive types to hold the larger /usr space. And it would still be nice to be able to do that from a small flash disk, etc. But you are right, I don’t think Linux ever did it correctly anyway….

  • Am 01.07.2012 07:40, schrieb Les Mikesell:
    [distinction between /bin and /usr/bin]

    The separation predates Unix networking. IIRC /usr/bin was already there on Unix Version 7 on the PDP-11, before Ethernet was even invented.

  • As mentioned on the busybox list a couple of years ago:

    Understanding the bin, sbin, usr/bin , usr/sbin split

    You know how Ken Thompson and Dennis Ritchie created Unix on a PDP-7 in 1969?
    Well around 1971 they upgraded to a PDP-11 with a pair of RK05 disk packs (1.5
    megabytes each) for storage.

    When the operating system grew too big to fit on the first RK05 disk pack (their root filesystem) they let it leak into the second one, which is where all the user home directories lived (which is why the mount was called /usr). They replicated all the OS directories under there (/bin, /sbin, /lib, /tmp…) and wrote files to those new directories because their original disk was out of space. When they got a third disk, they mounted it on /home and relocated all the user directories to there so the OS could consume all the space on both disks and grow to THREE WHOLE MEGABYTES (ooooh!).

    Of course they made rules about “when the system first boots, it has to come up enough to be able to mount the second disk on /usr, so don’t put things like the mount command /usr/bin or we’ll have a chicken and egg problem bringing the system up.” Fairly straightforward. Also fairly specific to v6 unix of 35
    years ago.

    The /bin vs /usr/bin split (and all the others) is an artifact of this, a
    1970’s implementation detail that got carried forward for decades by bureaucrats who never question _why_ they’re doing things. It stopped making any sense before Linux was ever invented, for multiple reasons:

    1) Early system bringup is the provice of initrd and initramfs, which deals with the “this file is needed before that file” issues. We’ve already _got_ a temporary system that boots the main system.

    2) shared libraries (introduced by the Berkeley guys) prevent you from independently upgrading the /lib and /usr/bin parts. They two partitions have to _match_ or they won’t work. This wasn’t the case in 1974, back then they had a certain level of independence because everything was statically linked.

    3) Cheap retail hard drives passed the 100 megabyte mark around 1990, and partition resizing software showed up somewhere around there (partition magic
    3.0 shipped in 1997).

    Of course once the split existed, some people made other rules to justify it.
    Root was for the OS stuff you got from upstream and /usr was for your site-
    local files. Then / was for the stuff you got from AT&T and /usr was for the stuff that your distro like IBM AIX or Dec Ultrix or SGI Irix added to it, and
    /usr/local was for your specific installation’s files. Then somebody decided
    /usr/local wasn’t a good place to install new packages, so let’s add /opt!
    I’m still waiting for /opt/local to show up…

    Of course given 30 years to fester, this split made some interesting distro-
    specific rules show up and go away again, such as “/tmp is cleared between reboots but /usr/tmp isn’t”. (Of course on Ubuntu /usr/tmp doesn’t exist and on Gentoo /usr/tmp is a symlink to /var/tmp which now has the “not cleared between reboots” rule. Yes all this predated tmpfs. It has to do with read-
    only root filesystems, /usr is always going to be read only in that case and
    /var is where your writable space is, / is _mostly_ read only except for bits of /etc which they tried to move to /var but really symlinking /etc to
    /var/etc happens more often than not…)

    Standards bureaucracies like the Linux Foundation (which consumed the Free
    Standards Group in its’ ever-growing accretion disk years ago) happily document and add to this sort of complexity without ever trying to understand why it was there in the first place. ‘Ken and Dennis leaked their OS into the equivalent of home because an RK05 disk pack on the PDP-11 was too small” goes whoosh over their heads.

    I’m pretty sure the busybox install just puts binaries wherever other versions of those binaries have historically gone. There’s no actual REASON for any of it anymore. Personally, I symlink /bin /sbin and /lib to their /usr equivalents on systems I put together. Embedded guys try to understand and simplify…


    GPLv3: as worthy a successor as The Phantom Menace, as timely as Duke Nukem
    Forever, and as welcome as New Coke.


    Giles Coochey, CCNA, CCNAS
    NetSecSpec Ltd
    +44 (0) 7983 877438

  • Which, of course, is a fallacy. Back when we did FSSTND the concept of initrd/initramfs wasn’t used. It still wasn’t too unusual to use rdev to set the root disk in the kernel image rather than pass it as a parameter via LILO. (I’m not sure when initrd became usable; it might have been around 2000. According to ramdisks only appeared in the kernel in 1996 and pivot_root in 2000. Thus initrd didn’t exist when FSSTND was written… in 1993/1994. Hardly “before Linux was ever invented”.)

    Irrelevant to the discussion. “Independence” of / and /usr was never an argument in the Linux space.

    Hmm, we did FSSTND in 1993/1994 timeframe. At this time, considering that for many people this was a 2nd OS dual-booted with DOS/Windows
    3, we dealt with smaller partitions spread potentially on the end of multiple disks that DOS couldn’t use (8Mb at the end of a 40Mb disk
    ‘cos of the 32Mb limit of the time. Woohoo, I had two of these disks!).

    That was never in FSSTND, and it wasn’t in the SVR4 definition documents
    (which I used a fair bit to help steer the FSSTND discussion).

    Basically, Rob is incorrect as to the Linux rationale. We kept the /
    and /usr split to be _consistent_ with existing practice and to allow small and hobbyist systems to be usable.

    _TODAY_ (where I have an MMC card in my wallet that’s almost 1000 times bigger than that old hard disk!) the rationale for a / and /usr split is a lot less. We _do_ commonly use ramdisk based boot systems, we _do_ have disk space to spare. If we were to redo FSSTND (er, FHS as it became)
    today then it’s likely that decisions like the / /usr split would have been decided differently.

    I’m not convinced a /XX -> /usr/XX symlink solution buys us very much (for cross-platform compatibility people will still have
    /usr/local/bin:/usr/bin:/bin as PATHs) but I’m also not convinced it’ll cause any real problems.

    If you want a historical look as to some of the reasons why we did what we did,

  • No it isn’t. There are still good reasons to not require unnecessary stuff on your boot device in spite of the fact that you’ve managed to find one way to work around the problem on one kind of hardware at a cost you can afford. And the the Linux layout pretty much forces everyone else to do the same.

  • Giles Coochey wrote:

    Um, nope. 100M drives weren’t “cheap” until well into the nineties.
    *Companies* could afford them by ’92 or ’93. ’90? I think it was the holiday season of ’89 that my bosses in the tiny, tiny software co. I
    worked for gave us presents, and mine include a *huge* 30M h/d (*way*
    bigger than the 20M I had), knowing they’d get use of it, with me working at home one weekends….

    Sun added /opt; I think Oracle used it, too, about the same time.

    At least here, my manager believes in /usr/local….



  • Your boot device can be (typically is) different to your root device.
    And even if you really meant “root” then this merger moves stuff _off_
    the root disk and puts it onto /usr. Properly done /usr can still be a separate partition, if desired; initrd just needs to mount it.

    So this proposal can make the root disk _smaller_ than it currently is.

  • You aren’t done booting until you complete the init scripts for runlevel 1. Having to have an extra copy of the kernel on yet another device to even get started seems wrong from a minimalist approach, and the need for sufficient ram for an initrd even more so. And you really should be able to mount /usr via nfs while retaining independent boot/diagnostic capability.

    What’s the problem with just following $PATH?

  • Typically your boot device is mounted at /boot (or, arguably, is the initrd) which isn’t affected by the merge.

    Personally, I think the merge is perfectly well reasoned, and I’m glad to see the system simplified. However, the wisdom of the merge is no longer a useful debate. It is coming, and separating / and /usr in your partitioning is no longer advised. You will lose functionality in future releases if you continue to do so.

  • There’s no extra kernel image needed. Kernel comes from /boot, just as today. You need to copy a few modules into the ramdisk so that drivers can be initialised, and you need a few user-space utilities (on RedHat 5
    these are mostly provided by “nash”; RedHat 6 is a lot more detailed in the initramfs image).

    Yes, this is one downside to the proposal; you can’t have a separate
    /usr partition on RAM limited machines (with RedHat you need, maybe, 4Mb of RAM for the initrd; with RedHat 6 that is gone up a lot). The memory should be free’d back once the switch to real-root has been made so it’s temporary but, yes, it is a requirement if you want separate
    / and /usr partitions.

    But, equally, these are likely to be storage limited machines as well (e.g embedded devices) so aren’t likely to want to waste space on partioning losses or will be using nfsroot, so this edge case isn’t a likely case.

    And if you _do_ have this pathological edge case then you don’t have to follow the Fedora standard; you’re gonna be going your own way, anyway!

    You can. Nothing in this proposal stops this.

    I’ve no idea. It’s not my proposal. I don’t see the _need_ to do it;
    indeed I said that most people will still set PATH to include bin and usr/bin simply ‘cos of cross-OS compatibility (AIX, HPUX, other Linuxen etc).

    I’m just saying that I don’t see it causing any true pain.

  • It may not – but it is always annoying to me when committees get together and move arbitrary things around just because they can. And even more so when they want to turn Linux, whose success is clearly based on the unix design into something that is not like unix.

  • You may not have noticed, but there is no longer any such thing. Red
    Hat’s init system never booted *through* runlevel 1, the way that some other Unix systems did.

    Even on older Red Hat systems, /usr is mounted before the runlevel 1
    scripts run. It’s mounted in rc.sysinit. By your logic, you aren’t done booting until you have both / and /usr mounted, so there’s no value in separating them.

    No part of the merge requires and additional copy of the kernel or initrd, and I’m not sure where your confusion on the subject originates.

    Prior to the merge, you could not reliably mount /usr from NFS, since it might not match the libraries in /. Merging / in to /usr actually makes an NFS root filesystem a supported configuration, which was not previously the case.

    Twice in this message, you’ve actually argued *for* merging the two without realizing it.

  • Yes, I noticed – and didn’t see the point in that either. Why copy the runlevel ‘look’ of sysV if you aren’t going to take advantage of it?

    Right. It was broken before. Which is mostly why I don’t have a lot of faith in the next round of changes either.

    I was arguing against the broken ways that the layout has diverged from the unix design concepts more than any specific thing. If this is going to make everything work again, great. This time for sure…

  • You are correct. I used to create and mail the tapes out with the software releases to the government and colleges. Ran V6/V7 on Dec
    PDP-11, early PC-compatible stuff (maybe System II used? Can’t recall for sure), 5B5/3B20 (AT&T designed hardware used when System III & V
    came available, IIRC) … when a big HD was 10MB (or even 5MB) and memory was 64K and processors where 8086, 80186, 80286, …

    Everything was small, highly unreliable compared to today. HDs were sliced up to try and ensure minimal damage when the inevitable crash occurred. Backup process recommended was “Tower of Hanoi” strategy on tape (which were also limited in size). A thing called “speed” was non-existent compared to today. So another reason for slicing was to reduce fsck times.

    The internet was not yet invented, but uucp and related provided
    “networking” capability.

    I gave a one day class to some *very* smart folks at DARPA and you know what happened next – “internet”.


  • Appreciate all the discussion, folks.

    Got some boxes that have some separate / and /usr, since we’re a /usr/local religion shop. (GRIN…) Someone long long ago, in a galaxy far, far away picked /usr as the split point, instead of /usr/local itself. So…

    Layers 8 and 9 of the OSI model, bite again… Religion and Politics.

    Guess we’ll be moving to /opt or /usr/local being the separate mountpoint. I’m sure this will be a happy internal discussion… hahaha…