26 thoughts on - Preferred Method Of Provisioning VM Images

• I would recommend the default be the libvirtd tools, as it tries to be hypervisor-agnostic, so it’s most portable. Also, it provides a good amount of flexibility for various install type, where images/docker is more restricted. Again, speaking about defaults/generalities.

  I would, of course, point to the other tools so that users know what exists and what might work best for their use case.

• Lars Kurth writes:

  Let me wear the hat of the user. The major hurdles were network setup, installing something in a vm, and the chaotic state the documentation is in.

  I knew before I started that network setup would be a PITA because years ago, I set up a VM for someone who didn’t have a 64bit system to compile a 64bit version of some software. The network setup being so ridiculously difficult has kept me from touching VMs ever again for years. It’s just too difficult and not worth the effort unless you’re really forced to do it.

  As a user, I’m used to get an ISO of an installer or of a life system, put that into a DVD drive or write it to an USB stick and to boot from that to do the installation. Why can’t I do that with xen?

  I wouldn’t want #4. I want to be able to just use the installer of whatever distribution I’m installing in a VM and a simple way of saying
  “run this in a VM and let me install”. Or perhaps I merely want to run a life system in a VM to try it out, with network access. What easy way is there to do that?

  I haven’t heared of #3.

  #2 and #1 lead to total confusion. Xen doesn’t have any way to use an installer ISO to install a VM. The documentation is confusing because it lacks clarity and refers to many versions of xen at the same time which suggest to do things differently so that you don’t know how you are supposed to do something. There is no documentation for CentOS
  about how to set up a VM.

  #2 and #1 use different ways and different formats of configuration files, and apparently #1 doesn’t support everything #2 does and brings about an undesirable overhead while apparently being a good choice for users who also use other things than xen and don’t want to learn different ways of managing VMs.

  When I use xen, I do not need #1, and it seems to make more sense to me to stick with #2.

  #2 needs clarity of documentation. I’ve got an installer ISO. The documentation needs to tell me straightforwardly what I need to do to get a VM running with this installer ISO. I don’t care about #1–4 at that point.

  It’s that simple. More details can be learned over time.

  Insofar the documentation needs to deal with different versions of xen, just split the documentation entirely. When I use version X, I do not want to read the documentation for versions A, B, C or Z. That’s confusing and I have to ask myself all the time which part of what I’m reading might apply to the software I’m using, and that makes reading the documentation very painful.

  It’s like you buy a new TV and it comes with 50000 pages of documentation because the manufacturer of the TV mentions everything for every model of TV they ever manufactured, rather than documenting no more than the TV you actually bought. Trying to figure out how to switch to a different channel or how to increase the volume takes at least two days each.

  For example: http://wiki.xen.org/wiki/Credit_Scheduler:

  “It is now the default scheduler in the xen-unstable trunk.”

  So is it the default in the version I’m using or what?

  “The SEDF and BVT schedulers are still optionally available but the plan of record is for them to be phased out and eventually removed.”

  What are those and have they been removed in the version I’m using? But then, it probably doesn’t matter what they are because they might have been removed anyway. So what is this information supposed to tell me??

  “A domain with a weight of 512 will get twice as much CPU as a domain with a weight of 256 on a contended host. Legal weights range from 1 to
  65535 and the default is 256.”

  Where would I change this default? What happens when I set a weight of
  100 or of 33? I can only guess that it might be rounded up or down to something like 128 or 64 or 32 or 64. Or perhaps actually floating point math is employed to compute the relations of the different weights? Probably not …

  “Schedule Rate Limiting (added in Xen 4.2)”

  That doesn’t belong there. It needs to go into the documentation for that particular version of xen.

  “Timeslice (added in Xen 4.2)
  […]
  # xl sched-credit -t [n]”

  dito

  “Usage

  The xm sched-credit command may be used to tune the per VM guest scheduler parameters.”

  So what am I supposed to use? xm or xl? Or something else, maybe virsh?

  I always admire the documentation exim has … As to your original question:

  I’d suggest to start with splitting the documentation into different versions, one version of documentation for each version of xen. Some way to get a VM running may then be discovered and improved upon.

  What actually is the way to do it? I found something that works, and it allows me to install Debian squeeze, from which then I upgrade to wheezy. For anything else, I don’t know.

• Wow… “chaotic state” :-O

  Don’t get me wrong. I know everything can always be improved, and documentation is –for free software in general, which well includes Xen– often in a position of needing _a_lot_ of improvement.

  All that being said, it would be helpful if you could be a bit more specific and help us actually improve it. I think you’ve said, in another thread, that you requested editor access to the wiki, but don’t know what status such request is in (provided I’m not mistaking you with someone else, in which case, sorry! :-P). Can you double check, so that we can see how to move forward and allow you to act instead of only complaining. :-D

  Wait, what? Who said you can’t do it? Of course you can! In fact, if you tried and it failed, please, report the bug within the appropriate channel, because a bug is what that is, nothing less! :-O

  However, I think we’re talking about something different here…

  I’m not sure I got what you mean with ‘I merely want to run a life system in a VM to try it out’. If you mean trying out Xen in a VM, then of course you need a nested-virt capable platform. If you have one, here’s the instructions of how to create a Xen liveCD. I’m afraid both the instructions and the liveCD are Fedora based, but it should not be a big deal to bias them toward CentOS:

  http://lists.xen.org/archives/html/xen-users/2014-06/msg00065.html

  I also don’t know much of #3, but at least #1 and #2, are both meant at doing exactly what you’re saying: “install this in a VM”, they only happen not to require install DVDs. I understand that DVDs are your preferred method, but that doesn’t make it the one _everyone_ prefer!

  AFAIUI, when talking about this kind of provisioning, what we’re interested is something a lot more automated and, as a consequence, a lot less interactive than an installer.

  Ok, now we’re talking! That’s what I meant when asking to ‘be specific’! :-)

  Good point, I’ll fix this.

  Ditto.

  I don’t agree. I think they do belnog. Of course, that does not mean we can’t link to them from a ‘Xen 4.2’ page.

  Personally, I don’t think that would be the optimal split, or something we need to do right now (which does not mean we should not improve docs in other ways). However, I think you’re way off the ‘original question’
  which, again, was about provisioning.

  Regards, Dario

• Networking can be confusing until it “clicks” then it all seems to fall into place, there is a certain amount of understanding of the overall picture of how bridge networking (or other types of networking) works that once you get that understanding it becomes a lot simpler. One thing that probably confuses you is that there are certain parts of the networking that are done by the dom0 distro, certain parts are done by Xen scripts and front and backend drivers, and some parts are done by the domu operating system. Understanding how those parts fit together and how different distros do the same thing but perhaps with different configs helps a lot to getting a grasp on the whole.

  Obviously better documentation would help with this, but like anything of this nature there is a learning curve that you simply have to get over before you can really become proficient.

  You can do that with Xen, KVM, and most other types of virtualization. The idea is that you need to block-attach the disk (or ISO file if you prefer not to use a physical disk) to the VM and then it can be mounted and used as install media (or any other type of media for that matter). This can be done from teh domain config file or from the xm (or xl)
  command line.

  Another quick note specifically for you. I have noticed in past messages that you seem to be having some confusion with xm vs xl commands and some explanation about that is probably in order to clear up that confusion. xm and xl are nearly identical commands to use, but xm is the “old” way and xl the “new” way. Basically put, xm interfaces with a daemon running in the dom0 (xend) which in turn interfaces with the hypervisor, while xl uses libraries to interface directly with the hypervisor and so eliminates the need for xend. You will see documentation refer to one or the other, sometimes almost interchangeably, because the two tools have almost identical usage, but what you should be aware of is that you really need to pick one and stick with it, Xen does not work well if you try to use xm for some commands and xl for others. Since you’re new I would recommend xl because xm is deprecated in newer versions of Xen, so if you want to future-proof your knowledge xl is the way to go. Also if you’re using xl then you should not be running the xend daemon, this is easily accomplished by using chkconfig and service to turn xend off.

  Peter

• This is not a complete list of the ways you can install a VM either. My personal preference is to manually create the filesystem for the VM and then install the OS core with yum. Then after tweaking some config files you can start up the VM and finish installing whatever else you want with yum as well.

  While I don’t think that this should be the recommended install method it might be worth mentioning and even giving a wiki page with some instructions on how to do it this way. Doing an install like this is actually very good for a newbie because you “get your hands dirty” and get a really good understanding of how yum works and the internals of the distro.

  Peter

• Hi!

  When you’re saying “this only works with xm (when xl is used, this can create some undefined behavior”, that is because of the particular version of libvirt (and Xen) is available in CentOS?

  I’m asking because, with current enough versions of both, situation should not be that bad, at least not as far as virt-install and VM
  provisioning are concerned… :-O

  Personally, I’m not sure whether this is the best solution, but I see it like something that should not be left in a ‘non-working’ state.

  Parhaps, a disadvantage I see in going for it as the default and recommended mechanism is that you really need:
  – a recent version of libvit to start with
  – a way to update libvirt quite often, to take advantage of new
  features being added

  Yep, and I went beyond that, and produced some spec files. This should be the latest one, although it’s for version 4.3.1 of xen-tools, so no, not the latest possible or the tip of the repo:
  http://dariof.fedorapeople.org/SPECS/xen-tools.spec

  My main problem with xen-tools is that it is kind of debian (and derivatives) biased, when it comes to building the guest file system. I.e., last time I tried, provisioning a Fedora or CentOS guest was possible, but not as straightforward as creating a debian one. That is not xen-tool’s fault, is the fact there were no good alternatives to debootstrap.

  However, it looks like rinse (one of said alternatives to debootstrap)
  is getting some steam again:
  https://gitorious.org/rinse

  so that may actually be worth a try.

  Also, I recently saw this in xen-tools-discuss mailing list:

  “anouncement : Xen-tools update for CentOS 6.5”
  http://xen-tools.org/pipermail/xen-tools-discuss/2014-May/001053.html

  Even if only for the title, it looks like something to investigate! :-P

  Talking a bit with Axel (xen-tools’ maintainer) at the time, the main issue in packaging subsequent versions of xen-tools for Fedora was going to be some dependencies from a bunch of perl module he wrote, and packaged himself for Debian, as separate packages. Nothing terrible, but that would mean making the RPMs for those modules as well.

  TBH, it is a while back, and I’m not sure what happened, whether went that route. This link, from the xen-tools-discuss email mentioned above:
  https://github.com/remsnet/xen-tools-rpmbuild

  seems to have (more) updated spec files. I haven’t tried them, though.

  Not commenting on this as I don’t know it.

  All that being said, and FWIW, I think I’d prefer #1, provided a recent enough version of Xen and Libvirt could be used.

  Regards, Dario

• Indeed it does!

  If you’re up for it, Xen wiki will be glad to host it! :-P

  We’ve got both CentOS and Fedora categories on the Wiki, both hosting various pages, guides, instructions, etc., but none of them (I think)
  goes through the procedure you mentioned:

  http://wiki.xen.org/wiki/Category:CentOS
  http://wiki.xen.org/wiki/Category:Fedora

  Feel free to add it! :-)

  Regards, Dario

• I’ve chosen the virt-install method on CentOS 5 precisely because it is like KVM. I was hoping it would fulfill the promise of being hypervisor agnostic. I’m hoping it continues to be available on future versions of CentOS with Xen.

  Though it is a waste of resources, I make all my virtual machines, Linux and MS Windows alike, fully virtualized. I can then move any of the VM’s with the same virt-install –import or virsh dumpxml/edit/virsh define process.

  When moving a VM, usually the only thing I have to do outside of virt-install/virt-manager is add , or , which can be done with virsh edit. I don’t know why some of my virtual servers need them and other don’t but I have higher priority things to think about.

  I’m the only technical support person and I don’t work 24/7. The graphical interface of virt-manager makes it possible for non-tech people to see what is running and see consoles to restart any misbehaving VM’s (usually MS Windows VM’s).

  I have completely eliminated my use of xm.

• @Peter, Would you mind sharing the process?
  (on the CentOS or Xen wikis OR share a link to a write-up online?)

  This piques my curiosity and I’d like to know more. ;-)
  Thanks!

  +1 :-D

• I would love to do a writeup on this, but my time is extremely limited right now. I’ll see what I can do.

  Peter
  —–BEGIN PGP SIGNATURE—

• Actually, what I’ll probably do is wait for CentOS 7 to drop and then do a writeup based on that. I’ll be wanting to set up a VM from it anyways and so then I’ll kill two birds with one stone.

  Peter
  —–BEGIN PGP SIGNATURE—

• The world needs documentation bounties.

• Dario Faggioli writes:

  It’s really difficult to be any specific because I have to piece together information from whatever sources are available through search engines and by trial and error.

  Perhaps take practical questions and try to answer them from the documentation you can find, for example:

  I have an installer for distribution X. How do I install the distribution in a VM?

  How do I make sure that a particular network interface I have in dom0
  will always show up as a particular network interface in domU?

  What is an xvda device, what are the impacts of using one instead of, for example, /dev/sda, and how are the devices named when I have several of them because after installing a VM, I want to give it access to another logical volume: xvda1, xvda2, etc., or xvdb, xvdc, or something else?

  Why does it say “I need more memory …” when trying to create a guest with xm? Can’t I do memory overcommittment? My dom0 has plenty of swap space.

  What are the effects of overcommitting CPUs? Should I rather limit the VMs tightly or give them CPUs so that they can be used as needed instead of being left idle?

  Do I need to load a particular module in a VM to be able to give it more memory with xm? What happens when I give a VM more memory and then take it away after a while?

  How do I make it so that memory is assigned to VMs dynamically up to their maximum allowed amounts, depending on how much memory a VM
  currently happens to need?

  Why gives “xenpm get-cpufreq-states” no output?

  Why do I get an entry “(XEN) physdev.c:168: dom0: wrong map_pirq type 3”
  in xm dmesg, and what is it supposed to tell me?

  When you assume you’re new to using xen and try to answer questions like this from whatever documentation you can find, you’ll probably see what I mean :)

  I think the request has been successful. I’m still working on getting everything set up and didn’t get to look into editing the wiki yet.

  Well, how do you do that? IIRC, I followed http://wiki.xen.org/wiki/Debian_Guest_Installation_Using_Debian_Installer, and it installs squeeze instead of wheezy. Apparently you require a special kernel and initrd.img to install something in a VM — so how do you install something else?

  No, I was assuming that I have a dom0 running and decide, for example, that I want to try out the live DVD of some Fedora spin in another VM.

  Oh I’ve seen this page — it wasn’t was I was looking for, though.

  An installer is merely what I find or have when I want to install a distribution. Suppose I’d want to try out Mint. I’d look at their website and download one of the versions. I’d burn a DVD or CD or write it to an USB stick and boot from that.

  With xen, what do I do instead? It seems I need a special version to get a VM running with it. Where do I get this special version?

  Wouldn’t that depend a lot on what you’re trying to do? Some people suggest to do a basic install and then to copy that.

  The documentation would be much easier to read without having to filter it for the version of xen that is being used. Why not just copy the current documentation when the next version is released and remove the cruft? Then you can simply say “it goes like this” instead of having to say “it might go like this but only if you’re using that version, and perhaps this way will be removed, but you might be using something that could be deprecated in the next version, so don’t use this but there are five other ways to do it …”.

  For things removed just say ‘removed, see feature X’, and for new things, mark them as ‘new in the current version’. This helps users of previous versions to catch up and saves new users from filtering the cruft.

  Hm, yes, I didn’t understand at first that this a different question, sorry.

• We ship a test/devel grade CentOS-6-x86_64-pv image ( well, its a qcow2
  image, should work for pvhvm as well, the fstab is label driven ).[1]

  The biggest problem in doing pre-baked images is the instance metadata. We need to find an easy way to get network settings into the instance and the root password ( or key ), and finally – in some cases, console redirection/setup, but i dont think the console is a deal breaker or a big deal. The network and access credentials however are.

  In a typical cloud environ this info would come from the cloud controller’s metadata service; on a typical virtualised setup though this becomes an issue ( and isnt really Xen specific ).

  We could work around this by making some assumptions, we could ‘own’
  dnsmasq and ensure that either libvirt is running and doing dhcp, otherwise we do the dhcp with some sane defaults, or we setup a script to ‘instantiate image’, which asks how the user wants to setup the instance ( pvhvm, hvm, pv ), the root password or key to use, and the network settings ( and if this is run on the dom0, we could even ask what bridge or device to connect with as well as the settings ).[2]

  Ofcourse, having these images pushed from here mean that clouds or virtualised environs that have metadata services are able to just-use the image as is, not needing any more tooling etc. And we can easily push monthly image updates and when things like heartbleed come around, there is a single place we need to update.

  – KB
  [1]: http://cloud.CentOS.org/CentOS/6/devel/

  [2] might need to pull in all of libguestfs to make the changes, which in turn has its own challenges if run inside a virtualised environ.

• Karanbir Singh writes:

  Wouldn’t you still need to configure the services running in each VM?

• Yes.

  I believe the idea here is to pre-configure as much as possible and complete the remaining configuration via network connectivity (ie: SSH).

• I didn’t follow this — virt-builder seems to run fine in dom0 with qemu, albeit a bit slow. Upstream is open to having patches for Xen bindings for the utility VM.

  Or is there something else I’m missing?

  -George

• SilverTip257 writes:

  How about some sort of package management that lets you define and configure the VM?

  This is currently done from “the inside”, i. e. when the VM is running, with whatever installer and package manager a distribution comes with.

  Why not do it from “the outside”, i. e. before the VM even exists, creating it in the process? Define some sort of API so that the same
  “VM creator tool” could be used with different distributions.

• Am 17.06.2014 19:31, schrieb lee:

  I believe you are talking about a bootstrapped minimal installation and than managing this via a config management tool like puppet, chef etc. ?

  Nevertheless I think this gets really off topic and out of scope.

• A couple of notes on virt-builder:

  * When I tried it, all of the images booted on Xen out-of-the-box except Fedora 19 and 20; I reported this to the list and the developers were very responsive, so now everything works. (The exception might be RHEL7-rc, but I think should be gone now.)
  * However, libguestfs has a massive dependency tree in RHEL7 for some reason; KB said that a “yum install libguestfs” wanted to install 178
  additional RPMs. A minimal C6 install is only 143 RPMs, so this more than doubles it. That makes this a bit less desirable as the primary method of getting a system up and running quickly.

  So it looks like we might want to recommend three potential paths we could recommend people to explore:

  1) For basic CentOS VMs, use a CentOS-provided cloud image, with our custom metadata tweaking script.

  2) For more versatile image set-up and manipulation (including other operating systems), use virt-builder.

  3) If you’re thinking about using libvirt anyway, use virt-install and install from installation media.

  Does that sound reasonable?

  #1 still has some issues we need to work out — particularly with sorting out the metadata. I’ll start a separate thread to talk about that.

  -George

• I just dived a little in virt-builder and here are my findings:

  1. It’s awesome!

  Am 18.06.2014 12:46, schrieb George Dunlap:

  2. Why create a custom script if maybe virt-builder can handle it?
  I don’t know if you can point virt-builder to other images than those on libguestfs.org, but this should be possible, include an CentOS
  cloud-image on libguestfs.org and just use virt-builder

  3. as a fallback if virt-builder is not available or you don’t got the images from libguestfs.org cached or you don’t have direct internet access: ok, for other purposes I guess virt-builder should be enough?

• If networking is already configured when a VM boots, how about a kernel parameter for a configuration server? If it is configured to grab an executable file, it would be very flexible.

  We use kickstart files to build our CentOS VM’s and physical boxes and then pull configuration changes from a server at regular intervals. I’m working on extending our existing internal intranetto keep history and to allow a single change to propagate to all relevant servers. Also would allow fail2ban to share it’s banned IP’s. Since this is specific to our internal systems, it can’t easily be exported, but most of it is pretty straight forward. It could almost be integrated into an existing CMS. Think WebMin but in a LISA (Large Installation System Administration) environment.

  A similar technique could be used for cloud machines.

• Sven Kieske writes:

  No, I’m talking about an “installer” which can be used to create and to configure a VM image. The resulting VM image would have all needed packages installed and be fully configured.

  Does “provisioning VM images” not involve their creation?

• You can point virt-builder to another repo; and in fact, the guys on the libguestfs development list said that them hosting their own template repo was meant to be a temporary measure, and that long-term they’d like to see distros hosting their own templates.

  But as to “why”, did you miss that “yum install libguestfs” wants to bring in an *additional* 178 packages, more than doubling the size of a base installation? :-)

  I think for a lot of people that will be fine; but I agree with KB, that we should also have a really lightweight way for people to just get VMs up and running.

  -George

• Building stripped OS images is an art form. In fact, if you want to get *really* dirty, don’t use disk images. Use a local chroot cage to install an OS image from an initial minimal tarball, and use tools like “kickstart” to pre-partition the disk and run theset steps..

  1) Configure disk.
  a( Use parted in CentOS 6, or fdisk in CentOS 5 to pre-align partitions on the 4096 byte block boundaries. All VM’s that might run on NetApp or other 4096 byte block drive back end storage needs this, and better safe than sorry.
  2) Obtain tarball, for example by downloading it into the swap partition if your installer doesn’t have a local copy on a DVD image or local NFS share.
  3) Configure /etc/fstab in the chroot cage. (Don’t even put one in the tarball, it’s not needed!)
  4) Configure /etc/syscontig/network-scripts/ifcfg-eth0 in the tarball, as needed, for network layout.
  a) Pay attention to /etc/udev.d/rules/70-persistent-net-rules, which will reset your network device numbering.
  b) It’s usually safest to just delete it for VM and server images.
  5) Run “grub-install” for the relevant bood disk, from inside the chroot cage, to make it bootable.
  6: Clear the installation tarball and run “mkswap” on the swap partition.

  I don’t know why most VM environments don’t use a separate disk image for swap, anyway, burying it inside your LVM setup makes it much more onerous to efficiently cut free or resize without disturbing the rest of your system. And it makes a thin-provisioned disk automatically much bulkier, instead just leave out swap from published OS disk images. But that’s a very separate issue.

  Now, when I first did this sort of thing, it was to shrink a corporate Red Hat 5.2 build onto an installable CD. (Not RHEL. .2 Red Hat 5.2)
  The original engineer had been using dd’d and gzipped disk images, of different sizes for different disk drives, and trying to put them on the same CD, and always starting over by zeroing the disk and putting new images on it to make any changes, then dd’ing and re-compressing the disk image. Including *swap*!!!!! The same sort of wasted disk, written but uncleared blocks, swap space, and other issues exist for VM images, and a lot of the resulting image size is entirely unneeded for OS installations. Getting the original OS image tarball can be pesky: I had to install an OS, reboot with a rescue CD, and tarball the contents of the disk partitions. I got it down so small it fit on a “buisness card” CD.

  But these days, guess what? “mock” already does most of this work for me!!! Take a good look at the stored initialization tarballs for
  “mock”, and they have *exactly* this sort of content. They have some extra components for RPM building, but you can enter the chroot cage and strip those right out, along with unnecessary log files, by staying at run level 1 while in the tarball. I applaud the “mock”
  authors for their work creating well defined workable chroot cages, and take other advantage of them for testing components.

  The initial setup takes some knowledge, and it’s no longer just “plug in this disk image and be done”, you need a working tool like a PXE
  environment and a selection of pre-configured kickstart files with
  ‘%pre” scripts, or an installer CD or DVD or USB image with a kickstart script set up, to make it efficient But it scales *really well*. I helped do 13,000 systems in one month this way, back in 2002, and the technique has proven invaluable for prototyping new, completely defined, completely reviewable OS images ever since.

• Am 18.06.2014 23:33, schrieb lee:

  Yes, this is e.g. something you can do with saltstack salt-virt or similar tools (especially the “fully configured” part). but you could also use virt-builder/-installer + cloud-init or various other tools.

  No, not necessary.